[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:05 dist-test-slave-xn4h krb5kdc[17325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:05 dist-test-slave-xn4h krb5kdc[17325](info): set up 2 sockets
Sep 24 15:59:05 dist-test-slave-xn4h krb5kdc[17325](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 15:59:06.917691 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:07 dist-test-slave-xn4h krb5kdc[17325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:07.103747 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34887
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:34887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:07.616489 17338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:07.617554 17338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:07.617651 17338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:07.625519 17338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:07.625644 17338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:07.625696 17338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:07.625746 17338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:07.625792 17338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:07.636824 17338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:34887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:07.640831 17338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:07.643689 17338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:07.675448 17346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:07.676071 17343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:07.676460 17344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:07.677644 17338 server_base.cc:1034] running on GCE node
I20260924 15:59:07.681617 17338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:07.683734 17338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:07.684986 17338 hybrid_clock.cc:648] HybridClock initialized: now 1790265547684890 us; error 91 us; skew 500 ppm
I20260924 15:59:07.686040 17338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:07 dist-test-slave-xn4h krb5kdc[17325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:07.689816 17338 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:07.691982 17338 webserver.cc:460] Webserver started at http://127.16.233.190:43243/ using document root <none> and password file <none>
I20260924 15:59:07.693720 17338 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:07.693903 17338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:07.694363 17338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:07.701941 17338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "76d59af551814d27acb92fea2a1acd33"
format_stamp: "Formatted at 2026-09-24 15:59:07 on dist-test-slave-xn4h"
server_key: "dfb98e30308eb846615e02235a2da86a"
server_key_iv: "f74823cf337bd3b901d5601819b63c2e"
server_key_version: "encryptionkey@0"
I20260924 15:59:07.702980 17338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "76d59af551814d27acb92fea2a1acd33"
format_stamp: "Formatted at 2026-09-24 15:59:07 on dist-test-slave-xn4h"
server_key: "dfb98e30308eb846615e02235a2da86a"
server_key_iv: "f74823cf337bd3b901d5601819b63c2e"
server_key_version: "encryptionkey@0"
I20260924 15:59:07.712258 17338 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.000s
I20260924 15:59:07.719920 17353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:07.721505 17338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 15:59:07.721798 17338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "76d59af551814d27acb92fea2a1acd33"
format_stamp: "Formatted at 2026-09-24 15:59:07 on dist-test-slave-xn4h"
server_key: "dfb98e30308eb846615e02235a2da86a"
server_key_iv: "f74823cf337bd3b901d5601819b63c2e"
server_key_version: "encryptionkey@0"
I20260924 15:59:07.722036 17338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:07.743285 17338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:07.748190 17338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:07.748512 17338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:07.783350 17338 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:34887
I20260924 15:59:07.783411 17406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:34887 every 8 connection(s)
I20260924 15:59:07.785130 17338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:07.790009 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 17338
I20260924 15:59:07.790426 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:07.792529 17318 external_mini_cluster.cc:1428] Setting key f593a41a1aa4926c4b74280970078240
I20260924 15:59:07.797104 17407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:07.815907 17407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33: Bootstrap starting.
I20260924 15:59:07.821946 17407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:07.824204 17407 log.cc:824] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:07.827946 17407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33: No bootstrap required, opened a new log
Sep 24 15:59:07 dist-test-slave-xn4h krb5kdc[17325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:07.844449 17407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d59af551814d27acb92fea2a1acd33" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34887 } }
I20260924 15:59:07.845011 17407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:07.845206 17407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d59af551814d27acb92fea2a1acd33, State: Initialized, Role: FOLLOWER
I20260924 15:59:07.846469 17407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d59af551814d27acb92fea2a1acd33" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34887 } }
I20260924 15:59:07.847035 17407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:07.847324 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:07.802458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.16.233.190:34887)
0924 15:59:07.803492 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:07.803504 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:07.807375 (+ 3871us) server_negotiation.cc:407] Connection header received
0924 15:59:07.819006 (+ 11631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:07.819036 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:07.819821 (+ 785us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:07.820355 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:07.824784 (+ 4429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:07.827039 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:07.828819 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:07.829973 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:07.830172 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:07.836497 (+ 6325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:07.836513 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:07.836593 (+ 80us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:07.836651 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:07.840968 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:07.841872 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:07.841881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:07.841890 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:07.842169 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:07.842707 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:07.842711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:07.842713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:07.844096 (+ 1383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:07.844446 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:07.845438 (+ 992us) server_negotiation.cc:299] Negotiation successful
0924 15:59:07.845878 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":418,"threads_started":1}
I20260924 15:59:07.847332 17407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:07.848606 17407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:07.851449 17407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d59af551814d27acb92fea2a1acd33" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34887 } }
I20260924 15:59:07.853204 17407 leader_election.cc:302] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d59af551814d27acb92fea2a1acd33; no voters:
I20260924 15:59:07.853844 17407 leader_election.cc:288] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:07.854004 17412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:07.856135 17412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [term 1 LEADER]: Becoming Leader. State: Replica: 76d59af551814d27acb92fea2a1acd33, State: Running, Role: LEADER
I20260924 15:59:07.856711 17412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d59af551814d27acb92fea2a1acd33" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34887 } }
I20260924 15:59:07.859494 17407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:07.875377 17414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d59af551814d27acb92fea2a1acd33. Latest consensus state: current_term: 1 leader_uuid: "76d59af551814d27acb92fea2a1acd33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d59af551814d27acb92fea2a1acd33" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34887 } } }
I20260924 15:59:07.875377 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d59af551814d27acb92fea2a1acd33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d59af551814d27acb92fea2a1acd33" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34887 } } }
I20260924 15:59:07.875957 17414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:07.875996 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33 [sys.catalog]: This master's current role is: LEADER
W20260924 15:59:07.883592 17427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:59:07.883807 17427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:59:07.884093 17428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:07.886525 17428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:07.897910 17428 catalog_manager.cc:1347] Generated new cluster ID: 7b74539374bd43299aa31506dc3f88ef
I20260924 15:59:07.898008 17428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:07.919627 17428 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:07.921133 17428 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:07.929817 17428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d59af551814d27acb92fea2a1acd33: Generated new TSK 0
I20260924 15:59:07.930900 17428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 15:59:07.979780 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:34887
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:08.398761 17433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:08.399048 17433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:08.399212 17433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:08.408242 17433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:08.408478 17433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:08.408758 17433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:08.420019 17433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:34887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:08.421393 17433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:08.422863 17433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:08.441918 17439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:08.442036 17438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:08.442688 17441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:08.443115 17433 server_base.cc:1034] running on GCE node
I20260924 15:59:08.444252 17433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:08.445622 17433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:08.446874 17433 hybrid_clock.cc:648] HybridClock initialized: now 1790265548446817 us; error 49 us; skew 500 ppm
I20260924 15:59:08.447309 17433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:08 dist-test-slave-xn4h krb5kdc[17325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265548, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:08.452387 17433 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 15:59:08.455258 17433 webserver.cc:460] Webserver started at http://127.16.233.129:43721/ using document root <none> and password file <none>
I20260924 15:59:08.456578 17433 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:08.456844 17433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:08.457357 17433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:08.462770 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "a1dbd97a6f4c45c8b9dbe718a6b07db8"
format_stamp: "Formatted at 2026-09-24 15:59:08 on dist-test-slave-xn4h"
server_key: "b97b3490a0a5e1444d1d07e3c5dee450"
server_key_iv: "b3bbac41d201bf34ac31ce8f50e86ecb"
server_key_version: "encryptionkey@0"
I20260924 15:59:08.464103 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "a1dbd97a6f4c45c8b9dbe718a6b07db8"
format_stamp: "Formatted at 2026-09-24 15:59:08 on dist-test-slave-xn4h"
server_key: "b97b3490a0a5e1444d1d07e3c5dee450"
server_key_iv: "b3bbac41d201bf34ac31ce8f50e86ecb"
server_key_version: "encryptionkey@0"
I20260924 15:59:08.471081 17433 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 15:59:08.475533 17448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:08.477246 17433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260924 15:59:08.477432 17433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "a1dbd97a6f4c45c8b9dbe718a6b07db8"
format_stamp: "Formatted at 2026-09-24 15:59:08 on dist-test-slave-xn4h"
server_key: "b97b3490a0a5e1444d1d07e3c5dee450"
server_key_iv: "b3bbac41d201bf34ac31ce8f50e86ecb"
server_key_version: "encryptionkey@0"
I20260924 15:59:08.477648 17433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:08.502233 17433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:08.506786 17433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:08.507110 17433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:08.508217 17433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:08.511724 17433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:08.511914 17433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:08.512075 17433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:08.512151 17433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:08.562161 17433 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:41497
I20260924 15:59:08.562222 17561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:41497 every 8 connection(s)
I20260924 15:59:08.563566 17433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:08.569867 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 17433
I20260924 15:59:08.570144 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:08.571054 17318 external_mini_cluster.cc:1428] Setting key 93511eba8a8fcb6e67372dc9eff4ce7a
Sep 24 15:59:08 dist-test-slave-xn4h krb5kdc[17325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265548, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
I20260924 15:59:08.591168 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:08.568557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:46319 (local address 127.16.233.190:34887)
0924 15:59:08.569031 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:08.569036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:08.570640 (+ 1604us) server_negotiation.cc:407] Connection header received
0924 15:59:08.573228 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:08.573232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:08.573298 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:08.573453 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:08.575088 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:08.576567 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:08.578295 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:08.579001 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:08.579217 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:08.583592 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:08.583598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:08.583601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:08.583637 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:08.587034 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:08.588079 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:08.588084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:08.588086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:08.588321 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:08.588949 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:08.588957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:08.588960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:08.589278 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:08.589423 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:08.590424 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0924 15:59:08.590710 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":272,"threads_started":1}
I20260924 15:59:08.592736 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:08.569269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:34887 (local address 127.16.233.129:46319)
0924 15:59:08.570426 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:08.570476 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:08.572881 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:08.573679 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:08.573689 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:08.574234 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:08.574878 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:08.574897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:08.576763 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:08.576768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:08.578133 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:08.578146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:08.579278 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:08.579288 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:08.579599 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:08.580476 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:08.580500 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:08.583314 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:08.587245 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:08.587257 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:08.587278 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:08.587898 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:08.588458 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:08.588464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:08.588468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:08.588712 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:08.589453 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:08.589460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:08.589967 (+ 507us) client_negotiation.cc:769] Sending connection context
0924 15:59:08.590227 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 15:59:08.590915 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":470,"threads_started":1}
I20260924 15:59:08.594481 17562 heartbeater.cc:347] Connected to a master server at 127.16.233.190:34887
I20260924 15:59:08.594815 17562 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:08.595736 17562 heartbeater.cc:511] Master 127.16.233.190:34887 requested a full tablet report, sending...
I20260924 15:59:08.600837 17370 ts_manager.cc:194] Registered new tserver with Master: a1dbd97a6f4c45c8b9dbe718a6b07db8 (127.16.233.129:41497)
I20260924 15:59:08.604761 17370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:46319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 15:59:08.617865 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:34887
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:09.028244 17568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:09.028564 17568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:09.028673 17568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:09.037101 17568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:09.037228 17568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:09.037345 17568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:09.047765 17568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:34887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:09.049093 17568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:09.050527 17568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:09.068737 17573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:09.068710 17574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:09.069043 17576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:09.069949 17568 server_base.cc:1034] running on GCE node
I20260924 15:59:09.070710 17568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:09.071844 17568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:09.073055 17568 hybrid_clock.cc:648] HybridClock initialized: now 1790265549073009 us; error 38 us; skew 500 ppm
I20260924 15:59:09.073350 17568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:09 dist-test-slave-xn4h krb5kdc[17325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:09.086031 17568 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 15:59:09.088027 17568 webserver.cc:460] Webserver started at http://127.16.233.130:35493/ using document root <none> and password file <none>
I20260924 15:59:09.088768 17568 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:09.088909 17568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:09.089251 17568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:09.092574 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "ab29f98337764315a6db2abc007472ed"
format_stamp: "Formatted at 2026-09-24 15:59:09 on dist-test-slave-xn4h"
server_key: "32d5b1acb693ae4e2a87c608313e15f0"
server_key_iv: "e2e114835ecd30d43972b0f8f956311b"
server_key_version: "encryptionkey@0"
I20260924 15:59:09.093567 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "ab29f98337764315a6db2abc007472ed"
format_stamp: "Formatted at 2026-09-24 15:59:09 on dist-test-slave-xn4h"
server_key: "32d5b1acb693ae4e2a87c608313e15f0"
server_key_iv: "e2e114835ecd30d43972b0f8f956311b"
server_key_version: "encryptionkey@0"
I20260924 15:59:09.098971 17568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 15:59:09.102584 17583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:09.103986 17568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:09.104184 17568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "ab29f98337764315a6db2abc007472ed"
format_stamp: "Formatted at 2026-09-24 15:59:09 on dist-test-slave-xn4h"
server_key: "32d5b1acb693ae4e2a87c608313e15f0"
server_key_iv: "e2e114835ecd30d43972b0f8f956311b"
server_key_version: "encryptionkey@0"
I20260924 15:59:09.104408 17568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:09.129576 17568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:09.134141 17568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:09.134683 17568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:09.135763 17568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:09.137230 17568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:09.137306 17568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:09.137391 17568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:09.137436 17568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:09.185056 17568 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:37297
I20260924 15:59:09.185153 17696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:37297 every 8 connection(s)
I20260924 15:59:09.186488 17568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:09.188809 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 17568
I20260924 15:59:09.189018 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:09.189613 17318 external_mini_cluster.cc:1428] Setting key 18ff9b869cb9846400adec221b143fda
Sep 24 15:59:09 dist-test-slave-xn4h krb5kdc[17325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 15:59:09.208022 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:09.189081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:45211 (local address 127.16.233.190:34887)
0924 15:59:09.189574 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:09.189581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:09.190751 (+ 1170us) server_negotiation.cc:407] Connection header received
0924 15:59:09.192636 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:09.192645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:09.192738 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:09.192935 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:09.194587 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.195759 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.197409 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.197830 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.197974 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:09.201767 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:09.201772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:09.201775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:09.201810 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:09.204575 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.205420 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.205428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.205436 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.205567 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.206072 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.206076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.206078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.206427 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:09.206587 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:09.207401 (+ 814us) server_negotiation.cc:299] Negotiation successful
0924 15:59:09.207630 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
I20260924 15:59:09.209383 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:09.189674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:34887 (local address 127.16.233.130:45211)
0924 15:59:09.190585 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:09.190632 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:09.192371 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:09.193151 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:09.193160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:09.193722 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:09.194413 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:09.194436 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.195946 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.195950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:09.197230 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:09.197243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.198068 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.198075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:09.198414 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:09.199212 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:09.199242 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:09.201501 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:09.204774 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:09.204783 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:09.204809 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:09.205281 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:09.205701 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:09.205705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:09.205709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:09.205934 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:09.206566 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:09.206574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:09.206970 (+ 396us) client_negotiation.cc:769] Sending connection context
0924 15:59:09.207295 (+ 325us) client_negotiation.cc:240] Negotiation successful
0924 15:59:09.207882 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":375,"threads_started":1}
I20260924 15:59:09.210896 17697 heartbeater.cc:347] Connected to a master server at 127.16.233.190:34887
I20260924 15:59:09.211262 17697 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:09.212157 17697 heartbeater.cc:511] Master 127.16.233.190:34887 requested a full tablet report, sending...
I20260924 15:59:09.213905 17370 ts_manager.cc:194] Registered new tserver with Master: ab29f98337764315a6db2abc007472ed (127.16.233.130:37297)
I20260924 15:59:09.215250 17370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:45211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 15:59:09.231782 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:34887
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:59:09.609045 17562 heartbeater.cc:503] Master 127.16.233.190:34887 was elected leader, sending a full tablet report...
W20260924 15:59:09.633119 17703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:09.633440 17703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:09.633540 17703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:09.641058 17703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:09.641189 17703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:09.641413 17703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:09.652720 17703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:34887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:09.654050 17703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:09.655385 17703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:09.671880 17708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:09.671943 17709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:09.671968 17711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:09.672922 17703 server_base.cc:1034] running on GCE node
I20260924 15:59:09.673606 17703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:09.674970 17703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:09.676281 17703 hybrid_clock.cc:648] HybridClock initialized: now 1790265549676141 us; error 61 us; skew 500 ppm
I20260924 15:59:09.676736 17703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:09 dist-test-slave-xn4h krb5kdc[17325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:09.680327 17703 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 15:59:09.681912 17703 webserver.cc:460] Webserver started at http://127.16.233.131:45105/ using document root <none> and password file <none>
I20260924 15:59:09.682706 17703 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:09.682818 17703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:09.683176 17703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:09.686900 17703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "6b9072cc9e6b4314b67d7dc1749ea789"
format_stamp: "Formatted at 2026-09-24 15:59:09 on dist-test-slave-xn4h"
server_key: "f6480245e243baba4d274d20247f917c"
server_key_iv: "c1b7c84cc3ca18c49ebf8944cd4d29ab"
server_key_version: "encryptionkey@0"
I20260924 15:59:09.687700 17703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "6b9072cc9e6b4314b67d7dc1749ea789"
format_stamp: "Formatted at 2026-09-24 15:59:09 on dist-test-slave-xn4h"
server_key: "f6480245e243baba4d274d20247f917c"
server_key_iv: "c1b7c84cc3ca18c49ebf8944cd4d29ab"
server_key_version: "encryptionkey@0"
I20260924 15:59:09.693251 17703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 15:59:09.701224 17718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:09.703018 17703 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.003s
I20260924 15:59:09.703243 17703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "6b9072cc9e6b4314b67d7dc1749ea789"
format_stamp: "Formatted at 2026-09-24 15:59:09 on dist-test-slave-xn4h"
server_key: "f6480245e243baba4d274d20247f917c"
server_key_iv: "c1b7c84cc3ca18c49ebf8944cd4d29ab"
server_key_version: "encryptionkey@0"
I20260924 15:59:09.703482 17703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:09.738724 17703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:09.743557 17703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:09.743887 17703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:09.744956 17703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:09.746384 17703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:09.746491 17703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:09.746663 17703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:09.746739 17703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:09.791883 17703 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:46421
I20260924 15:59:09.791923 17830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:46421 every 8 connection(s)
I20260924 15:59:09.793306 17703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:09.802656 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 17703
I20260924 15:59:09.802973 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:09.803557 17318 external_mini_cluster.cc:1428] Setting key dc62286fc8699090670d670a0e55bb56
Sep 24 15:59:09 dist-test-slave-xn4h krb5kdc[17325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:09.814486 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:09.796028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:49209 (local address 127.16.233.190:34887)
0924 15:59:09.796518 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:09.796524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:09.797519 (+ 995us) server_negotiation.cc:407] Connection header received
0924 15:59:09.799397 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:09.799402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:09.799472 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:09.799586 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:09.801146 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.801982 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.803781 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.804183 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.804328 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:09.809005 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:09.809010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:09.809013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:09.809050 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:09.811633 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.812337 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.812340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.812342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.812474 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.812873 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.812876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.812877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.813144 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:09.813260 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:09.813950 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 15:59:09.814120 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":223,"threads_started":1}
I20260924 15:59:09.815474 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:09.796619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:34887 (local address 127.16.233.131:49209)
0924 15:59:09.797347 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:09.797386 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:09.799116 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:09.799827 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:09.799836 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:09.800364 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:09.800951 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:09.800969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.802134 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.802138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:09.803586 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:09.803600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.804421 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.804428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:09.804688 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:09.805471 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:09.805513 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:09.808794 (+ 3281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:09.811772 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:09.811779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:09.811792 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:09.812206 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:09.812568 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:09.812571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:09.812573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:09.812765 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:09.813266 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:09.813272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:09.813611 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 15:59:09.813847 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 15:59:09.814250 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":277,"threads_started":1}
I20260924 15:59:09.816846 17831 heartbeater.cc:347] Connected to a master server at 127.16.233.190:34887
I20260924 15:59:09.817171 17831 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:09.817925 17831 heartbeater.cc:511] Master 127.16.233.190:34887 requested a full tablet report, sending...
I20260924 15:59:09.819718 17370 ts_manager.cc:194] Registered new tserver with Master: 6b9072cc9e6b4314b67d7dc1749ea789 (127.16.233.131:46421)
I20260924 15:59:09.820714 17370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:49209
I20260924 15:59:09.821185 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:59:09 dist-test-slave-xn4h krb5kdc[17325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:59:09 dist-test-slave-xn4h krb5kdc[17325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:09.871315 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:09.852977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.16.233.129:41497)
0924 15:59:09.854067 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:09.854075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:09.854099 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:59:09.854188 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:09.854194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:09.854425 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:09.854659 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:09.856229 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.857814 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.859173 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.859812 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.859954 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:09.864762 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:09.864777 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:09.864819 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:09.864862 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:09.868021 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.868834 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.868842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.868845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.868990 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.869454 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.869460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.869464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.869858 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:09.870207 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:09.870622 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 15:59:09.870922 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":536,"threads_started":1}
W20260924 15:59:09.872668 17476 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38952
Sep 24 15:59:09 dist-test-slave-xn4h krb5kdc[17325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:59:09 dist-test-slave-xn4h krb5kdc[17325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:09.919550 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:09.904886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38966 (local address 127.16.233.129:41497)
0924 15:59:09.905142 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:09.905150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:09.905281 (+ 131us) server_negotiation.cc:407] Connection header received
0924 15:59:09.905438 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:09.905442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:09.905551 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:09.905639 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:09.907090 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.908004 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.909384 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.909810 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.909907 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:09.913792 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:09.913797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:09.913800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:09.913838 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:09.916501 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.917222 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.917226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.917229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.917461 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.917894 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.917897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.917899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.918220 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:09.918551 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:09.918873 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 15:59:09.919073 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 15:59:09.921233 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 17433
I20260924 15:59:09.935667 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:09.923702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.16.233.129:41497)
0924 15:59:09.923870 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:09.923875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:09.924393 (+ 518us) server_negotiation.cc:407] Connection header received
0924 15:59:09.924502 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:09.924506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:09.924567 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:09.924647 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:09.925851 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.926679 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.927981 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:09.928419 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:09.928523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:09.930872 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:09.930877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:09.930880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:09.930913 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:09.933389 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.933997 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.934001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.934004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.934135 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:09.934550 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:09.934554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:09.934556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:09.934830 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:09.935016 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:09.935251 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 15:59:09.935411 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 15:59:09.936486 17557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:10.020417 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 17433
I20260924 15:59:10.037436 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 17568
Sep 24 15:59:10 dist-test-slave-xn4h krb5kdc[17325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 15:59:10.053565 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:10.038503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52078 (local address 127.16.233.130:37297)
0924 15:59:10.039022 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:10.039029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:10.039055 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:59:10.039116 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:10.039123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:10.039330 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:10.039452 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:10.040668 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:10.041699 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:10.043024 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:10.043592 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:10.043706 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:10.047782 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:10.047794 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:10.047810 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:10.047857 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:10.050775 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:10.051441 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:10.051448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:10.051450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:10.051578 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:10.051952 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:10.051955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:10.051958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:10.052322 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:10.052555 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:10.052890 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 15:59:10.053215 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":253,"threads_started":1}
I20260924 15:59:10.054975 17692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:10.141552 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 17568
I20260924 15:59:10.157956 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 17703
Sep 24 15:59:10 dist-test-slave-xn4h krb5kdc[17325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 15:59:10.174513 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:10.159000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.16.233.131:46421)
0924 15:59:10.159696 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:10.159703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:10.159733 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:59:10.159794 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:10.159801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:10.160013 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:10.160151 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:10.161513 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:10.162706 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:10.164010 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:10.164628 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:10.164727 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:10.169026 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:10.169038 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:10.169052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:10.169089 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:10.172006 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:10.172651 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:10.172657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:10.172660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:10.172769 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:10.173082 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:10.173085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:10.173087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:10.173411 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:10.173646 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:10.173854 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 15:59:10.174051 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":371,"threads_started":1}
I20260924 15:59:10.175839 17826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:10.258426 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 17703
I20260924 15:59:10.274680 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 17338
I20260924 15:59:10.275878 17401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:10.378228 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 17338
2026-09-24T15:59:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4581 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:10 dist-test-slave-xn4h krb5kdc[17853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:10 dist-test-slave-xn4h krb5kdc[17853](info): set up 2 sockets
Sep 24 15:59:10 dist-test-slave-xn4h krb5kdc[17853](info): commencing operation
krb5kdc: starting...
W20260924 15:59:11.467799 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:11 dist-test-slave-xn4h krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:11.642161 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:40503
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:11.960779 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:11.961045 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:11.961215 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:11.968844 17866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:11.968962 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:11.969038 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:11.969139 17866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:11.969218 17866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:11.978943 17866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:40503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:40503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:11.980221 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:11.981498 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:11.993043 17874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:11.993160 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:11.993131 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:11.994065 17866 server_base.cc:1034] running on GCE node
I20260924 15:59:11.994796 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:11.996119 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:11.997364 17866 hybrid_clock.cc:648] HybridClock initialized: now 1790265551997291 us; error 46 us; skew 500 ppm
I20260924 15:59:11.997718 17866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:11 dist-test-slave-xn4h krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265551, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:12.000730 17866 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:12.002085 17866 webserver.cc:460] Webserver started at http://127.16.233.190:44645/ using document root <none> and password file <none>
I20260924 15:59:12.002799 17866 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:12.002945 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:12.003297 17866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:12.006108 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "26f0fd41d55444629fc1085dd85c69ab"
format_stamp: "Formatted at 2026-09-24 15:59:11 on dist-test-slave-xn4h"
server_key: "2df2922269dbf7a478684b2d448d13db"
server_key_iv: "b90428eb693fb5700ee277b90a306b6a"
server_key_version: "encryptionkey@0"
I20260924 15:59:12.006906 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "26f0fd41d55444629fc1085dd85c69ab"
format_stamp: "Formatted at 2026-09-24 15:59:11 on dist-test-slave-xn4h"
server_key: "2df2922269dbf7a478684b2d448d13db"
server_key_iv: "b90428eb693fb5700ee277b90a306b6a"
server_key_version: "encryptionkey@0"
I20260924 15:59:12.011797 17866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 15:59:12.014978 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:12.016119 17866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 15:59:12.016310 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "26f0fd41d55444629fc1085dd85c69ab"
format_stamp: "Formatted at 2026-09-24 15:59:11 on dist-test-slave-xn4h"
server_key: "2df2922269dbf7a478684b2d448d13db"
server_key_iv: "b90428eb693fb5700ee277b90a306b6a"
server_key_version: "encryptionkey@0"
I20260924 15:59:12.016490 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:12.030206 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:12.034469 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:12.034756 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:12.053010 17866 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:40503
I20260924 15:59:12.053040 17932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:40503 every 8 connection(s)
I20260924 15:59:12.054550 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:12.058281 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 17866
I20260924 15:59:12.058504 17933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:12.058544 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:12.059314 17318 external_mini_cluster.cc:1428] Setting key 07d8b80843f1dd8e524261076ea739f1
I20260924 15:59:12.073211 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab: Bootstrap starting.
Sep 24 15:59:12 dist-test-slave-xn4h krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:12.076910 17933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:12.078213 17933 log.cc:824] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:12.081614 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab: No bootstrap required, opened a new log
I20260924 15:59:12.083295 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:12.061321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.16.233.190:40503)
0924 15:59:12.062093 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:12.062104 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:12.062145 (+ 41us) server_negotiation.cc:407] Connection header received
0924 15:59:12.063748 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:12.063767 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:12.064138 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:12.064469 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:12.066400 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:12.067630 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:12.069320 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:12.070012 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:12.070150 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:12.074808 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:12.074821 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:12.074840 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:12.074894 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:12.078704 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:12.079556 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:12.079562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:12.079568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:12.079796 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:12.080210 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:12.080215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:12.080218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:12.080897 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:12.081176 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:12.081588 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 15:59:12.082008 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":366,"threads_started":1}
I20260924 15:59:12.089519 17933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f0fd41d55444629fc1085dd85c69ab" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40503 } }
I20260924 15:59:12.089833 17933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:12.089910 17933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f0fd41d55444629fc1085dd85c69ab, State: Initialized, Role: FOLLOWER
I20260924 15:59:12.090431 17933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f0fd41d55444629fc1085dd85c69ab" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40503 } }
I20260924 15:59:12.090646 17933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:12.090727 17933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:12.090835 17933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:12.092819 17933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f0fd41d55444629fc1085dd85c69ab" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40503 } }
I20260924 15:59:12.093240 17933 leader_election.cc:302] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26f0fd41d55444629fc1085dd85c69ab; no voters:
I20260924 15:59:12.093781 17933 leader_election.cc:288] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:12.093922 17939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:12.095043 17939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [term 1 LEADER]: Becoming Leader. State: Replica: 26f0fd41d55444629fc1085dd85c69ab, State: Running, Role: LEADER
I20260924 15:59:12.095530 17939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f0fd41d55444629fc1085dd85c69ab" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40503 } }
I20260924 15:59:12.095927 17933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:12.098680 17941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26f0fd41d55444629fc1085dd85c69ab. Latest consensus state: current_term: 1 leader_uuid: "26f0fd41d55444629fc1085dd85c69ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f0fd41d55444629fc1085dd85c69ab" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40503 } } }
I20260924 15:59:12.099092 17941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:12.098285 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26f0fd41d55444629fc1085dd85c69ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f0fd41d55444629fc1085dd85c69ab" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40503 } } }
I20260924 15:59:12.099923 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab [sys.catalog]: This master's current role is: LEADER
W20260924 15:59:12.108141 17955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:59:12.108309 17955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:59:12.108443 17948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:12.110085 17948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:12.118173 17948 catalog_manager.cc:1347] Generated new cluster ID: 6d78c2184998497a8c0859af2fb550cf
I20260924 15:59:12.118285 17948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:12.145177 17948 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:12.146708 17948 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:12.154008 17948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26f0fd41d55444629fc1085dd85c69ab: Generated new TSK 0
I20260924 15:59:12.154906 17948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 15:59:12.203672 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:40503
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:12.519670 17960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:12.519936 17960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:12.520110 17960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:12.527407 17960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:12.527505 17960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:12.527645 17960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:12.537292 17960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:40503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:12.538708 17960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:12.540025 17960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:12.553165 17967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:12.553403 17966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:12.553381 17969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:12.554069 17960 server_base.cc:1034] running on GCE node
I20260924 15:59:12.554672 17960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:12.555546 17960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:12.556737 17960 hybrid_clock.cc:648] HybridClock initialized: now 1790265552556681 us; error 52 us; skew 500 ppm
I20260924 15:59:12.556998 17960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:12 dist-test-slave-xn4h krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:12.560106 17960 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 15:59:12.561568 17960 webserver.cc:460] Webserver started at http://127.16.233.129:38087/ using document root <none> and password file <none>
I20260924 15:59:12.562274 17960 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:12.562470 17960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:12.562785 17960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:12.565598 17960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "804c09a507d04191ae8c4dc58d8a5002"
format_stamp: "Formatted at 2026-09-24 15:59:12 on dist-test-slave-xn4h"
server_key: "385024b303f7e1cf5a1955caa7304dc2"
server_key_iv: "fad5085e5c54f7ad1305caedfed67342"
server_key_version: "encryptionkey@0"
I20260924 15:59:12.566450 17960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "804c09a507d04191ae8c4dc58d8a5002"
format_stamp: "Formatted at 2026-09-24 15:59:12 on dist-test-slave-xn4h"
server_key: "385024b303f7e1cf5a1955caa7304dc2"
server_key_iv: "fad5085e5c54f7ad1305caedfed67342"
server_key_version: "encryptionkey@0"
I20260924 15:59:12.571605 17960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 15:59:12.575012 17976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:12.576298 17960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:12.576467 17960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "804c09a507d04191ae8c4dc58d8a5002"
format_stamp: "Formatted at 2026-09-24 15:59:12 on dist-test-slave-xn4h"
server_key: "385024b303f7e1cf5a1955caa7304dc2"
server_key_iv: "fad5085e5c54f7ad1305caedfed67342"
server_key_version: "encryptionkey@0"
I20260924 15:59:12.576664 17960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:12.623255 17960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:12.627117 17960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:12.627427 17960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:12.628383 17960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:12.629678 17960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:12.629802 17960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:12.629930 17960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:12.629976 17960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:12.662824 17960 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:38503
I20260924 15:59:12.662914 18089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:38503 every 8 connection(s)
I20260924 15:59:12.664145 17960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:12.670781 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 17960
I20260924 15:59:12.671008 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:12.671630 17318 external_mini_cluster.cc:1428] Setting key 127a0e9929ddcbe570337fe08d1a67e8
Sep 24 15:59:12 dist-test-slave-xn4h krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:12.686388 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:12.666622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:44735 (local address 127.16.233.190:40503)
0924 15:59:12.667038 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:12.667043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:12.667995 (+ 952us) server_negotiation.cc:407] Connection header received
0924 15:59:12.669825 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:12.669831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:12.669933 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:12.670052 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:12.671696 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:12.672579 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:12.674808 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:12.675189 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:12.675284 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:12.679931 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:12.679936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:12.679940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:12.679997 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:12.682837 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:12.683841 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:12.683844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:12.683847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:12.683959 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:12.684452 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:12.684455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:12.684457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:12.684805 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:12.684918 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:12.685886 (+ 968us) server_negotiation.cc:299] Negotiation successful
0924 15:59:12.686070 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":202,"threads_started":1}
I20260924 15:59:12.688004 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:12.667180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:40503 (local address 127.16.233.129:44735)
0924 15:59:12.667838 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:12.667876 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:12.669558 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:12.670281 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:12.670289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:12.670868 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:12.671524 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:12.671542 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:12.672779 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:12.672783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:12.674644 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:12.674663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:12.675424 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:12.675433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:12.675782 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:12.676512 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:12.676547 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:12.679725 (+ 3178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:12.682971 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:12.682983 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:12.683005 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:12.683719 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:12.684066 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:12.684074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:12.684080 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:12.684339 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:12.684947 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:12.684958 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:12.685439 (+ 481us) client_negotiation.cc:769] Sending connection context
0924 15:59:12.685780 (+ 341us) client_negotiation.cc:240] Negotiation successful
0924 15:59:12.686452 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":266,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 15:59:12.689584 18090 heartbeater.cc:347] Connected to a master server at 127.16.233.190:40503
I20260924 15:59:12.689889 18090 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:12.690655 18090 heartbeater.cc:511] Master 127.16.233.190:40503 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
I20260924 15:59:12.692929 17898 ts_manager.cc:194] Registered new tserver with Master: 804c09a507d04191ae8c4dc58d8a5002 (127.16.233.129:38503)
I20260924 15:59:12.695109 17898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:44735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 15:59:12.714179 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:40503
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:13.031960 18096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:13.032228 18096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:13.032325 18096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:13.039605 18096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:13.039721 18096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:13.039834 18096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:13.049393 18096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:40503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:13.050647 18096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:13.051898 18096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:13.066638 18101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:13.066599 18102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:13.067034 18104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:13.067703 18096 server_base.cc:1034] running on GCE node
I20260924 15:59:13.068378 18096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:13.070567 18096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:13.071799 18096 hybrid_clock.cc:648] HybridClock initialized: now 1790265553071738 us; error 64 us; skew 500 ppm
I20260924 15:59:13.072139 18096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:13 dist-test-slave-xn4h krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:13.075631 18096 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 15:59:13.077054 18096 webserver.cc:460] Webserver started at http://127.16.233.130:32909/ using document root <none> and password file <none>
I20260924 15:59:13.077726 18096 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:13.077872 18096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:13.078181 18096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:13.081284 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "696f5f0de27b4d1cb6047e1955b4db5f"
format_stamp: "Formatted at 2026-09-24 15:59:13 on dist-test-slave-xn4h"
server_key: "d4c65639dd6fc116b3094fee3641a1df"
server_key_iv: "ca5f2fea96f5195abab687d2415a76cf"
server_key_version: "encryptionkey@0"
I20260924 15:59:13.082041 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "696f5f0de27b4d1cb6047e1955b4db5f"
format_stamp: "Formatted at 2026-09-24 15:59:13 on dist-test-slave-xn4h"
server_key: "d4c65639dd6fc116b3094fee3641a1df"
server_key_iv: "ca5f2fea96f5195abab687d2415a76cf"
server_key_version: "encryptionkey@0"
I20260924 15:59:13.086899 18096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 15:59:13.090121 18111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:13.091410 18096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:13.091581 18096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "696f5f0de27b4d1cb6047e1955b4db5f"
format_stamp: "Formatted at 2026-09-24 15:59:13 on dist-test-slave-xn4h"
server_key: "d4c65639dd6fc116b3094fee3641a1df"
server_key_iv: "ca5f2fea96f5195abab687d2415a76cf"
server_key_version: "encryptionkey@0"
I20260924 15:59:13.091787 18096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:13.105293 18096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:13.109103 18096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:13.109390 18096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:13.110284 18096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:13.111598 18096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:13.111708 18096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:13.111852 18096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:13.111929 18096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:13.145581 18096 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:44071
I20260924 15:59:13.145619 18224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:44071 every 8 connection(s)
I20260924 15:59:13.146924 18096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:13.150866 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 18096
I20260924 15:59:13.151067 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:13.151664 17318 external_mini_cluster.cc:1428] Setting key feec7c13f745eb3c992365c41c6b8bf5
Sep 24 15:59:13 dist-test-slave-xn4h krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 15:59:13.169497 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.149477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:44621 (local address 127.16.233.190:40503)
0924 15:59:13.149659 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:13.149663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:13.150883 (+ 1220us) server_negotiation.cc:407] Connection header received
0924 15:59:13.153190 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:13.153196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:13.153286 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:13.153416 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:13.155522 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.156190 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.158730 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.159447 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.159601 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.163469 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:13.163473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:13.163477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:13.163509 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:13.166214 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:13.167161 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:13.167165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:13.167167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:13.167285 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:13.167794 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:13.167798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:13.167800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:13.168058 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:13.168166 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:13.169080 (+ 914us) server_negotiation.cc:299] Negotiation successful
0924 15:59:13.169252 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 15:59:13.170647 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.149882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:40503 (local address 127.16.233.130:44621)
0924 15:59:13.150672 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:13.150734 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:13.152857 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:13.153679 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:13.153692 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:13.154480 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:13.155324 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.155354 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.156348 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.156356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.158591 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.158608 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.159656 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.159661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.159857 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.160560 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:13.160583 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:13.163227 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:13.166385 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:13.166395 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:13.166414 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:13.167035 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:13.167421 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:13.167426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:13.167430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:13.167677 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:13.168196 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:13.168206 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:13.168672 (+ 466us) client_negotiation.cc:769] Sending connection context
0924 15:59:13.168991 (+ 319us) client_negotiation.cc:240] Negotiation successful
0924 15:59:13.169431 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":265,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
I20260924 15:59:13.172180 18225 heartbeater.cc:347] Connected to a master server at 127.16.233.190:40503
I20260924 15:59:13.172518 18225 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:13.173233 18225 heartbeater.cc:511] Master 127.16.233.190:40503 requested a full tablet report, sending...
I20260924 15:59:13.174702 17898 ts_manager.cc:194] Registered new tserver with Master: 696f5f0de27b4d1cb6047e1955b4db5f (127.16.233.130:44071)
I20260924 15:59:13.175491 17898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:44621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 15:59:13.194092 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:40503
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:13.529832 18230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:13.530210 18230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:13.530340 18230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:13.537657 18230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:13.537778 18230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:13.537896 18230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:13.548502 18230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:40503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:13.549901 18230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:13.551362 18230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:13.565413 18235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:13.565655 18236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:13.565829 18238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:13.566946 18230 server_base.cc:1034] running on GCE node
I20260924 15:59:13.567605 18230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:13.568624 18230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:13.569856 18230 hybrid_clock.cc:648] HybridClock initialized: now 1790265553569807 us; error 48 us; skew 500 ppm
I20260924 15:59:13.570186 18230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:13 dist-test-slave-xn4h krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:13.573886 18230 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 15:59:13.575464 18230 webserver.cc:460] Webserver started at http://127.16.233.131:41729/ using document root <none> and password file <none>
I20260924 15:59:13.576172 18230 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:13.576316 18230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:13.576640 18230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:13.579550 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "da06515c331d48979fd7c9e41afd7622"
format_stamp: "Formatted at 2026-09-24 15:59:13 on dist-test-slave-xn4h"
server_key: "65a07e32ec47264a660cfa924fe73ca8"
server_key_iv: "c4cf2d26f85ed448dfdcb14be4aaa6bc"
server_key_version: "encryptionkey@0"
I20260924 15:59:13.580439 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "da06515c331d48979fd7c9e41afd7622"
format_stamp: "Formatted at 2026-09-24 15:59:13 on dist-test-slave-xn4h"
server_key: "65a07e32ec47264a660cfa924fe73ca8"
server_key_iv: "c4cf2d26f85ed448dfdcb14be4aaa6bc"
server_key_version: "encryptionkey@0"
I20260924 15:59:13.585609 18230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 15:59:13.589533 18245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:13.591092 18230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:13.591274 18230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "da06515c331d48979fd7c9e41afd7622"
format_stamp: "Formatted at 2026-09-24 15:59:13 on dist-test-slave-xn4h"
server_key: "65a07e32ec47264a660cfa924fe73ca8"
server_key_iv: "c4cf2d26f85ed448dfdcb14be4aaa6bc"
server_key_version: "encryptionkey@0"
I20260924 15:59:13.591535 18230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:13.611632 18230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:13.615602 18230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:13.615836 18230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:13.616730 18230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:13.618023 18230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:13.618096 18230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:13.618182 18230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:13.618227 18230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:13.663820 18230 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:41715
I20260924 15:59:13.663827 18358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:41715 every 8 connection(s)
I20260924 15:59:13.665419 18230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:13.672953 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 18230
I20260924 15:59:13.673174 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:13.673714 17318 external_mini_cluster.cc:1428] Setting key 4f8a5418c66d0c604c26d0b865cd1682
Sep 24 15:59:13 dist-test-slave-xn4h krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:13.687935 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.669319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:56743 (local address 127.16.233.190:40503)
0924 15:59:13.669525 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:13.669530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:13.670792 (+ 1262us) server_negotiation.cc:407] Connection header received
0924 15:59:13.673289 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:13.673296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:13.673413 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:13.673504 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:13.675766 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.676909 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.679048 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.679463 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.679557 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.682893 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:13.682899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:13.682902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:13.682932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:13.685203 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:13.685830 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:13.685834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:13.685836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:13.685951 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:13.686365 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:13.686369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:13.686371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:13.686624 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:13.686732 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:13.687369 (+ 637us) server_negotiation.cc:299] Negotiation successful
0924 15:59:13.687595 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 15:59:13.688948 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.669790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:40503 (local address 127.16.233.131:56743)
0924 15:59:13.670608 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:13.670649 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:13.672850 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:13.673814 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:13.673828 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:13.674566 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:13.675549 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.675579 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.677097 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.677104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.678914 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.678927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.679657 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.679662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.679853 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.680538 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:13.680565 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:13.682694 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:13.685321 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:13.685327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:13.685339 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:13.685729 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:13.686041 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:13.686044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:13.686046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:13.686211 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:13.686737 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:13.686742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:13.687013 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 15:59:13.687226 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 15:59:13.687651 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":278,"threads_started":1}
I20260924 15:59:13.690407 18359 heartbeater.cc:347] Connected to a master server at 127.16.233.190:40503
I20260924 15:59:13.690692 18359 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:13.691387 18359 heartbeater.cc:511] Master 127.16.233.190:40503 requested a full tablet report, sending...
I20260924 15:59:13.692785 17898 ts_manager.cc:194] Registered new tserver with Master: da06515c331d48979fd7c9e41afd7622 (127.16.233.131:41715)
I20260924 15:59:13.693562 17898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:56743
I20260924 15:59:13.698417 18090 heartbeater.cc:503] Master 127.16.233.190:40503 was elected leader, sending a full tablet report...
I20260924 15:59:13.701568 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:59:13.724352 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.710848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.16.233.190:40503)
0924 15:59:13.711091 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:13.711100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:13.711515 (+ 415us) server_negotiation.cc:407] Connection header received
0924 15:59:13.711653 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:13.711657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:13.711718 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:13.711799 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:13.713263 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.713963 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.715303 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.715706 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.715798 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.718556 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:13.718560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:13.718563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:13.718596 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:13.721935 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:13.722593 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:13.722596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:13.722598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:13.722782 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:13.723096 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:13.723099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:13.723101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:13.723400 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:13.723575 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:13.723853 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 15:59:13.724007 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 15:59:13.734814 17898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:59:13.749814 17898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:13.790132 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.771362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:44071 (local address 127.0.0.1:37484)
0924 15:59:13.772171 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:13.772205 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:13.772341 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:13.780540 (+ 8199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:13.780546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:13.780595 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:13.780952 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.780966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.783374 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.783378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.785239 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.785259 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.788464 (+ 3205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.788472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.788802 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.788884 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 15:59:13.789025 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 15:59:13.789332 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":390,"threads_started":1}
I20260924 15:59:13.790495 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.771558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.16.233.130:44071)
0924 15:59:13.773650 (+ 2092us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:13.773659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:13.773692 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:59:13.773777 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:13.773783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:13.773957 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:13.774168 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:13.781446 (+ 7278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.782774 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.785426 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.787804 (+ 2378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.788002 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.788177 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:13.789176 (+ 999us) server_negotiation.cc:299] Negotiation successful
0924 15:59:13.789466 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1947,"thread_start_us":1834,"threads_started":1}
I20260924 15:59:13.795781 18160 tablet_service.cc:1467] Processing CreateTablet for tablet 3131dc141dbf48c18708daed23a5ea00 (DEFAULT_TABLE table=test-table [id=562e868de58d4a1b841f3c87ab7c881c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:13.797555 18160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131dc141dbf48c18708daed23a5ea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:13.801343 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.773045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:41715 (local address 127.0.0.1:51652)
0924 15:59:13.786404 (+ 13359us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:13.786423 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:13.786544 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:13.788994 (+ 2450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:13.789000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:13.789031 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:13.789620 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.789633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.792088 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.792097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.798788 (+ 6691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.798802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.800517 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.800530 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.800701 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.800739 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:59:13.800833 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 15:59:13.800982 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13185,"thread_start_us":22994,"threads_started":1}
I20260924 15:59:13.802521 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.774448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:38503 (local address 127.0.0.1:34472)
0924 15:59:13.792350 (+ 17902us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:13.792369 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:13.792489 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:13.795334 (+ 2845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:13.795339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:13.795365 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:13.795629 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.795640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.797663 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.797668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.799456 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:13.799484 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.801695 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.801701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:13.801909 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.801944 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 15:59:13.802035 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 15:59:13.802186 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17799,"thread_start_us":11597,"threads_started":1}
I20260924 15:59:13.802886 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.774761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.16.233.129:38503)
0924 15:59:13.783886 (+ 9125us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:13.783898 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:13.792496 (+ 8598us) server_negotiation.cc:407] Connection header received
0924 15:59:13.792640 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:13.792652 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:13.792986 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:13.793291 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:13.795819 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.797449 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.799755 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.801228 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.801441 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.801593 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:13.802129 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 15:59:13.802369 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8902,"thread_start_us":10718,"threads_started":1}
I20260924 15:59:13.803241 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:13.773560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.16.233.131:41715)
0924 15:59:13.787604 (+ 14044us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:13.787614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:13.787681 (+ 67us) server_negotiation.cc:407] Connection header received
0924 15:59:13.787817 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:13.787827 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:13.788116 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:13.788460 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:13.789889 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.791924 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.799012 (+ 7088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:13.800304 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:13.800517 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:13.800730 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:13.800871 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 15:59:13.801026 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13790,"thread_start_us":13500,"threads_started":1}
I20260924 15:59:13.806926 18024 tablet_service.cc:1467] Processing CreateTablet for tablet 3131dc141dbf48c18708daed23a5ea00 (DEFAULT_TABLE table=test-table [id=562e868de58d4a1b841f3c87ab7c881c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:13.807291 18294 tablet_service.cc:1467] Processing CreateTablet for tablet 3131dc141dbf48c18708daed23a5ea00 (DEFAULT_TABLE table=test-table [id=562e868de58d4a1b841f3c87ab7c881c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:13.808260 18024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131dc141dbf48c18708daed23a5ea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:13.808888 18294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131dc141dbf48c18708daed23a5ea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:13.815268 18377 tablet_bootstrap.cc:492] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f: Bootstrap starting.
I20260924 15:59:13.819994 18377 tablet_bootstrap.cc:654] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:13.823236 18377 log.cc:824] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:13.831907 18380 tablet_bootstrap.cc:492] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622: Bootstrap starting.
I20260924 15:59:13.831977 18378 tablet_bootstrap.cc:492] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002: Bootstrap starting.
I20260924 15:59:13.835418 18377 tablet_bootstrap.cc:492] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f: No bootstrap required, opened a new log
I20260924 15:59:13.835587 18380 tablet_bootstrap.cc:654] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:13.835875 18377 ts_tablet_manager.cc:1397] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.006s
I20260924 15:59:13.836441 18378 tablet_bootstrap.cc:654] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:13.837049 18380 log.cc:824] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:13.838644 18378 log.cc:824] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:13.842865 18380 tablet_bootstrap.cc:492] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622: No bootstrap required, opened a new log
I20260924 15:59:13.843155 18380 ts_tablet_manager.cc:1397] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260924 15:59:13.843508 18378 tablet_bootstrap.cc:492] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002: No bootstrap required, opened a new log
I20260924 15:59:13.843984 18378 ts_tablet_manager.cc:1397] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260924 15:59:13.848423 18377 raft_consensus.cc:348] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } }
I20260924 15:59:13.848922 18377 raft_consensus.cc:374] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:13.849023 18377 raft_consensus.cc:729] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696f5f0de27b4d1cb6047e1955b4db5f, State: Initialized, Role: FOLLOWER
I20260924 15:59:13.849825 18377 consensus_queue.cc:260] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } }
I20260924 15:59:13.851271 18225 heartbeater.cc:503] Master 127.16.233.190:40503 was elected leader, sending a full tablet report...
I20260924 15:59:13.852078 18377 ts_tablet_manager.cc:1428] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260924 15:59:13.856918 18380 raft_consensus.cc:348] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } }
I20260924 15:59:13.857403 18380 raft_consensus.cc:374] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:13.857502 18380 raft_consensus.cc:729] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da06515c331d48979fd7c9e41afd7622, State: Initialized, Role: FOLLOWER
I20260924 15:59:13.858222 18378 raft_consensus.cc:348] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } }
I20260924 15:59:13.858274 18380 consensus_queue.cc:260] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } }
I20260924 15:59:13.858692 18378 raft_consensus.cc:374] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:13.858791 18378 raft_consensus.cc:729] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804c09a507d04191ae8c4dc58d8a5002, State: Initialized, Role: FOLLOWER
I20260924 15:59:13.859488 18378 consensus_queue.cc:260] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } }
I20260924 15:59:13.860481 18359 heartbeater.cc:503] Master 127.16.233.190:40503 was elected leader, sending a full tablet report...
I20260924 15:59:13.861668 18380 ts_tablet_manager.cc:1428] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260924 15:59:13.864110 18378 ts_tablet_manager.cc:1428] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002: Time spent starting tablet: real 0.020s user 0.017s sys 0.000s
W20260924 15:59:13.898864 18226 tablet.cc:2369] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:59:13.916718 18091 tablet.cc:2369] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:59:13.917840 18360 tablet.cc:2369] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:59:14.043895 18384 raft_consensus.cc:482] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:14.044245 18384 raft_consensus.cc:504] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } }
I20260924 15:59:14.046126 18384 leader_election.cc:288] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696f5f0de27b4d1cb6047e1955b4db5f (127.16.233.130:44071), 804c09a507d04191ae8c4dc58d8a5002 (127.16.233.129:38503)
I20260924 15:59:14.051760 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.046495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:44601 (local address 127.16.233.130:44071)
0924 15:59:14.046728 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:14.046735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:14.046791 (+ 56us) server_negotiation.cc:407] Connection header received
0924 15:59:14.047109 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:14.047115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:14.047191 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:14.047293 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:14.047672 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.048393 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.049955 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.050823 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.050988 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.051061 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:14.051377 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 15:59:14.051501 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 15:59:14.051779 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.046378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:44071 (local address 127.16.233.131:44601)
0924 15:59:14.046713 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:14.046734 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:14.046906 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:14.047315 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:14.047319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:14.047348 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:14.047530 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:14.047539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.048525 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.048532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:14.049797 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:14.049810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.050980 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.050986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:14.051208 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.051234 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 15:59:14.051338 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 15:59:14.051482 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1}
I20260924 15:59:14.052958 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131dc141dbf48c18708daed23a5ea00" candidate_uuid: "da06515c331d48979fd7c9e41afd7622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" is_pre_election: true
I20260924 15:59:14.053568 18180 raft_consensus.cc:2454] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da06515c331d48979fd7c9e41afd7622 in term 0.
I20260924 15:59:14.054428 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.046826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:45015 (local address 127.16.233.129:38503)
0924 15:59:14.047056 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:14.047062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:14.047503 (+ 441us) server_negotiation.cc:407] Connection header received
0924 15:59:14.047710 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:14.047714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:14.047770 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:14.047864 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:14.048568 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.049633 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.051872 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.053451 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.053623 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.053711 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:14.054015 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 15:59:14.054152 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 15:59:14.054514 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.046610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:38503 (local address 127.16.233.131:45015)
0924 15:59:14.047399 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:14.047414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:14.047571 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:14.047886 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:14.047890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:14.047915 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:14.048378 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:14.048390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.049865 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.049869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:14.051679 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:14.051691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.053650 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.053656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:14.053840 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.053874 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 15:59:14.053964 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 15:59:14.054095 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":46,"thread_start_us":448,"threads_started":1}
I20260924 15:59:14.054399 18246 leader_election.cc:302] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696f5f0de27b4d1cb6047e1955b4db5f, da06515c331d48979fd7c9e41afd7622; no voters:
I20260924 15:59:14.054893 18384 raft_consensus.cc:2781] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:14.055035 18384 raft_consensus.cc:482] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:14.055119 18384 raft_consensus.cc:3037] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:14.055703 18045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131dc141dbf48c18708daed23a5ea00" candidate_uuid: "da06515c331d48979fd7c9e41afd7622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804c09a507d04191ae8c4dc58d8a5002" is_pre_election: true
I20260924 15:59:14.056207 18045 raft_consensus.cc:2454] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da06515c331d48979fd7c9e41afd7622 in term 0.
I20260924 15:59:14.058290 18384 raft_consensus.cc:504] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } }
I20260924 15:59:14.059026 18384 leader_election.cc:288] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [CANDIDATE]: Term 1 election: Requested vote from peers 696f5f0de27b4d1cb6047e1955b4db5f (127.16.233.130:44071), 804c09a507d04191ae8c4dc58d8a5002 (127.16.233.129:38503)
I20260924 15:59:14.059510 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131dc141dbf48c18708daed23a5ea00" candidate_uuid: "da06515c331d48979fd7c9e41afd7622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696f5f0de27b4d1cb6047e1955b4db5f"
I20260924 15:59:14.059540 18045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131dc141dbf48c18708daed23a5ea00" candidate_uuid: "da06515c331d48979fd7c9e41afd7622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804c09a507d04191ae8c4dc58d8a5002"
I20260924 15:59:14.059684 18180 raft_consensus.cc:3037] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:14.059742 18045 raft_consensus.cc:3037] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:14.061823 18180 raft_consensus.cc:2454] T 3131dc141dbf48c18708daed23a5ea00 P 696f5f0de27b4d1cb6047e1955b4db5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da06515c331d48979fd7c9e41afd7622 in term 1.
I20260924 15:59:14.062354 18246 leader_election.cc:302] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696f5f0de27b4d1cb6047e1955b4db5f, da06515c331d48979fd7c9e41afd7622; no voters:
I20260924 15:59:14.063422 18384 raft_consensus.cc:2781] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:14.063549 18045 raft_consensus.cc:2454] T 3131dc141dbf48c18708daed23a5ea00 P 804c09a507d04191ae8c4dc58d8a5002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da06515c331d48979fd7c9e41afd7622 in term 1.
I20260924 15:59:14.064004 18384 raft_consensus.cc:686] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [term 1 LEADER]: Becoming Leader. State: Replica: da06515c331d48979fd7c9e41afd7622, State: Running, Role: LEADER
I20260924 15:59:14.064642 18384 consensus_queue.cc:237] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } }
I20260924 15:59:14.074793 17897 catalog_manager.cc:5495] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 reported cstate change: term changed from 0 to 1, leader changed from <none> to da06515c331d48979fd7c9e41afd7622 (127.16.233.131). New cstate: current_term: 1 leader_uuid: "da06515c331d48979fd7c9e41afd7622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696f5f0de27b4d1cb6047e1955b4db5f" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 44071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da06515c331d48979fd7c9e41afd7622" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 41715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "804c09a507d04191ae8c4dc58d8a5002" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 38503 } health_report { overall_health: UNKNOWN } } }
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:14.171696 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.141350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.16.233.129:38503)
0924 15:59:14.141545 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:14.141553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:14.142256 (+ 703us) server_negotiation.cc:407] Connection header received
0924 15:59:14.142517 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:14.142522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:14.142594 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:14.142713 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:14.144769 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.145548 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.147051 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.147505 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.147647 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.165906 (+ 18259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:14.165926 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:14.165942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:14.165983 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:14.168733 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.169480 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.169488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.169490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.169630 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.170032 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.170037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.170039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.170472 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:14.170789 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:14.171093 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 15:59:14.171316 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265554, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265554, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:14.217980 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.203516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.16.233.129:38503)
0924 15:59:14.203756 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:14.203764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:14.204056 (+ 292us) server_negotiation.cc:407] Connection header received
0924 15:59:14.204273 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:14.204278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:14.204344 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:14.204470 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:14.206043 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.206826 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.208086 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.208504 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.208646 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.212468 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:14.212475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:14.212482 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:14.212549 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:14.215686 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.216314 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.216318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.216320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.216447 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.216747 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.216750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.216752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.217023 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:14.217207 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:14.217415 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 15:59:14.217593 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:14.264648 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.251084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.16.233.129:38503)
0924 15:59:14.251335 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:14.251342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:14.251609 (+ 267us) server_negotiation.cc:407] Connection header received
0924 15:59:14.251769 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:14.251780 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:14.251861 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:14.251950 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:14.253269 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.253981 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.255238 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.255649 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.255742 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.259448 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:14.259453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:14.259456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:14.259496 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:14.261828 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.262468 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.262472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.262474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.262586 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.262902 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.262905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.262907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.263330 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:14.263660 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:14.263828 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 15:59:14.264206 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 15:59:14.266554 18004 tablet_service.cc:2908] Rejecting scan request for tablet 3131dc141dbf48c18708daed23a5ea00: Uninitialized: clean time has not yet been initialized
I20260924 15:59:14.270690 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 17960
I20260924 15:59:14.283522 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.272025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.16.233.129:38503)
0924 15:59:14.272202 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:14.272206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:14.272471 (+ 265us) server_negotiation.cc:407] Connection header received
0924 15:59:14.272610 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:14.272614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:14.272685 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:14.272764 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:14.273887 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.274654 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.276065 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.276473 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.276573 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.278761 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:14.278765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:14.278769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:14.278801 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:14.281412 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.281973 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.281977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.281979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.282090 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.282432 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.282435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.282437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.282679 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:14.282882 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:14.283095 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 15:59:14.283280 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 15:59:14.284363 18085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:14.387173 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 17960
I20260924 15:59:14.404640 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 18096
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 15:59:14.418661 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.405371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.16.233.130:44071)
0924 15:59:14.405549 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:14.405554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:14.405610 (+ 56us) server_negotiation.cc:407] Connection header received
0924 15:59:14.405752 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:14.405756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:14.405822 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:14.405915 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:14.407124 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.407827 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.409019 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.409385 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.409478 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.413263 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:14.413277 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:14.413292 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:14.413330 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:14.416370 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.416957 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.416963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.416965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.417088 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.417383 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.417387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.417390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.417717 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:14.417936 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:14.418182 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 15:59:14.418401 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 15:59:14.419330 18220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:14.468122 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.467611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:38503 (local address 127.16.233.131:38385)
0924 15:59:14.467806 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:14.467949 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:38503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 15:59:14.468479 18248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.129:38503: connect: Connection refused (error 111)
I20260924 15:59:14.470064 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.469665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:38503 (local address 127.16.233.131:47659)
0924 15:59:14.469828 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:14.469919 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:38503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 15:59:14.470274 18248 consensus_peers.cc:487] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 -> Peer 804c09a507d04191ae8c4dc58d8a5002 (127.16.233.129:38503): Couldn't send request to peer 804c09a507d04191ae8c4dc58d8a5002. Status: Network error: Client connection negotiation failed: client connection to 127.16.233.129:38503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 15:59:14.508541 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 18096
I20260924 15:59:14.525372 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 18230
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 15:59:14.536044 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.535552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:44071 (local address 127.16.233.131:42329)
0924 15:59:14.535735 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:14.535819 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.130:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 15:59:14.537276 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.536779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:44071 (local address 127.16.233.131:40227)
0924 15:59:14.537009 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:14.537100 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.130:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 15:59:14.537477 18246 consensus_peers.cc:487] T 3131dc141dbf48c18708daed23a5ea00 P da06515c331d48979fd7c9e41afd7622 -> Peer 696f5f0de27b4d1cb6047e1955b4db5f (127.16.233.130:44071): Couldn't send request to peer 696f5f0de27b4d1cb6047e1955b4db5f. Status: Network error: Client connection negotiation failed: client connection to 127.16.233.130:44071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 15:59:14.540757 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:14.526175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.16.233.131:41715)
0924 15:59:14.526709 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:14.526713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:14.526732 (+ 19us) server_negotiation.cc:407] Connection header received
0924 15:59:14.526785 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:14.526789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:14.526850 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:14.526941 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:14.528355 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.529071 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.530353 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:14.530755 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:14.530846 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:14.534970 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:14.534984 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:14.535000 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:14.535056 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:14.538465 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.539042 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.539048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.539051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.539186 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:14.539494 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:14.539497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:14.539499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:14.539804 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:14.540002 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:14.540250 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 15:59:14.540412 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":271,"threads_started":1}
I20260924 15:59:14.541344 18354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:14.631389 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 18230
I20260924 15:59:14.647959 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 17866
I20260924 15:59:14.649178 17928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:14.757730 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 17866
2026-09-24T15:59:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4381 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[18409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[18409](info): set up 2 sockets
Sep 24 15:59:14 dist-test-slave-xn4h krb5kdc[18409](info): commencing operation
krb5kdc: starting...
W20260924 15:59:15.851502 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:15 dist-test-slave-xn4h krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:16.038791 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41823
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:41823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:16.359102 18422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:16.359468 18422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:16.359640 18422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:16.366926 18422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:16.367087 18422 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:59:16.367167 18422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:16.367272 18422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:16.367369 18422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:16.367456 18422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:16.377291 18422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:41823
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:16.378688 18422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:16.380044 18422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:16.392006 18427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:16.391927 18428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:16.391964 18430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:16.392912 18422 server_base.cc:1034] running on GCE node
I20260924 15:59:16.393671 18422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:16.395205 18422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:16.396426 18422 hybrid_clock.cc:648] HybridClock initialized: now 1790265556396341 us; error 57 us; skew 500 ppm
I20260924 15:59:16.396770 18422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:16 dist-test-slave-xn4h krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265556, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:16.399991 18422 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:16.401569 18422 webserver.cc:460] Webserver started at http://127.16.233.190:35937/ using document root <none> and password file <none>
I20260924 15:59:16.402240 18422 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:16.402390 18422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:16.402684 18422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:16.405754 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe"
format_stamp: "Formatted at 2026-09-24 15:59:16 on dist-test-slave-xn4h"
server_key: "7b3ff1104d9a7d963d798601e40a636b"
server_key_iv: "780fc2fb2e254265efe72d7cf1580ea1"
server_key_version: "encryptionkey@0"
I20260924 15:59:16.406582 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe"
format_stamp: "Formatted at 2026-09-24 15:59:16 on dist-test-slave-xn4h"
server_key: "7b3ff1104d9a7d963d798601e40a636b"
server_key_iv: "780fc2fb2e254265efe72d7cf1580ea1"
server_key_version: "encryptionkey@0"
I20260924 15:59:16.411618 18422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 15:59:16.414918 18437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:16.416198 18422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:16.416360 18422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe"
format_stamp: "Formatted at 2026-09-24 15:59:16 on dist-test-slave-xn4h"
server_key: "7b3ff1104d9a7d963d798601e40a636b"
server_key_iv: "780fc2fb2e254265efe72d7cf1580ea1"
server_key_version: "encryptionkey@0"
I20260924 15:59:16.416599 18422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:16.445840 18422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:16.450475 18422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:16.450780 18422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:16.472795 18422 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:41823
I20260924 15:59:16.472816 18499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:41823 every 8 connection(s)
I20260924 15:59:16.474175 18422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:16.475795 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 18422
I20260924 15:59:16.476039 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:16.476846 17318 external_mini_cluster.cc:1428] Setting key 5115db3a67b057bc1753ac2bce204941
I20260924 15:59:16.478926 18500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 15:59:16 dist-test-slave-xn4h krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:16.492285 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe: Bootstrap starting.
I20260924 15:59:16.495687 18500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:16.496809 18500 log.cc:824] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:16.499468 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:16.478889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.16.233.190:41823)
0924 15:59:16.479673 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:16.479684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:16.479723 (+ 39us) server_negotiation.cc:407] Connection header received
0924 15:59:16.481325 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:16.481346 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:16.481738 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:16.482066 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:16.483801 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:16.485096 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:16.486811 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:16.487441 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:16.487582 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:16.492203 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:16.492211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:16.492224 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:16.492259 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:16.495295 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:16.496083 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:16.496090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:16.496102 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:16.496277 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:16.496678 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:16.496681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:16.496684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:16.497256 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:16.497601 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:16.498008 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 15:59:16.498499 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":400,"threads_started":1}
I20260924 15:59:16.499897 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe: No bootstrap required, opened a new log
I20260924 15:59:16.507941 18500 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41823 } }
I20260924 15:59:16.508311 18500 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:16.508430 18500 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf3eba8a1be4cfb9dc72fd60158adfe, State: Initialized, Role: FOLLOWER
I20260924 15:59:16.509032 18500 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41823 } }
I20260924 15:59:16.509264 18500 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:16.509392 18500 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:16.509559 18500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:16.512112 18500 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41823 } }
I20260924 15:59:16.512631 18500 leader_election.cc:302] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf3eba8a1be4cfb9dc72fd60158adfe; no voters:
I20260924 15:59:16.513176 18500 leader_election.cc:288] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:16.513309 18505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:16.514000 18505 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [term 1 LEADER]: Becoming Leader. State: Replica: ccf3eba8a1be4cfb9dc72fd60158adfe, State: Running, Role: LEADER
I20260924 15:59:16.514575 18505 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41823 } }
I20260924 15:59:16.515327 18500 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:16.517829 18507 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf3eba8a1be4cfb9dc72fd60158adfe. Latest consensus state: current_term: 1 leader_uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41823 } } }
I20260924 15:59:16.518235 18507 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:16.518767 18506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf3eba8a1be4cfb9dc72fd60158adfe" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41823 } } }
I20260924 15:59:16.519044 18506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:16.519366 18514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:16.525269 18514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:16.534473 18514 catalog_manager.cc:1347] Generated new cluster ID: 8ad0b96660504b6f8a9b9766054299da
I20260924 15:59:16.534598 18514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:16.547571 18514 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:16.549415 18514 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:16.564688 18514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe: Generated new TSK 0
I20260924 15:59:16.565897 18514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 15:59:16.618758 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41823
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:16.947260 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:16.947552 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:16.947716 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:16.955179 18526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:16.955296 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:16.955446 18526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:59:16.955583 18526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:16.965579 18526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.190:41823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:16.966969 18526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:16.968386 18526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:16.981789 18532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:16.981740 18531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:16.981848 18534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:16.982610 18526 server_base.cc:1034] running on GCE node
I20260924 15:59:16.983173 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:16.984045 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:16.985263 18526 hybrid_clock.cc:648] HybridClock initialized: now 1790265556985192 us; error 36 us; skew 500 ppm
I20260924 15:59:16.985577 18526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:16 dist-test-slave-xn4h krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265556, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:16.988915 18526 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 15:59:16.990489 18526 webserver.cc:460] Webserver started at http://127.16.233.129:37181/ using document root <none> and password file <none>
I20260924 15:59:16.991194 18526 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:16.991321 18526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:16.991677 18526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:16.994668 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "0292be8689d04d4ab112eafa591b6d70"
format_stamp: "Formatted at 2026-09-24 15:59:16 on dist-test-slave-xn4h"
server_key: "7ee21c1ecc8ab582d3388a37116374b9"
server_key_iv: "c71ea4551182aff2163539d0e4110313"
server_key_version: "encryptionkey@0"
I20260924 15:59:16.995604 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "0292be8689d04d4ab112eafa591b6d70"
format_stamp: "Formatted at 2026-09-24 15:59:16 on dist-test-slave-xn4h"
server_key: "7ee21c1ecc8ab582d3388a37116374b9"
server_key_iv: "c71ea4551182aff2163539d0e4110313"
server_key_version: "encryptionkey@0"
I20260924 15:59:17.001320 18526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 15:59:17.004760 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:17.006103 18526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 15:59:17.006266 18526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "0292be8689d04d4ab112eafa591b6d70"
format_stamp: "Formatted at 2026-09-24 15:59:16 on dist-test-slave-xn4h"
server_key: "7ee21c1ecc8ab582d3388a37116374b9"
server_key_iv: "c71ea4551182aff2163539d0e4110313"
server_key_version: "encryptionkey@0"
I20260924 15:59:17.006547 18526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:17.018733 18526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:17.022866 18526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:17.023134 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:17.025658 18526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:17.025772 18526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:17.025909 18526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:17.025986 18526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:59:17 dist-test-slave-xn4h krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265556, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:17.048918 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:17.026441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:45209 (local address 127.16.233.190:41823)
0924 15:59:17.026896 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:17.026900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:17.028224 (+ 1324us) server_negotiation.cc:407] Connection header received
0924 15:59:17.030208 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:17.030213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:17.030300 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:17.030429 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:17.032010 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.032896 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.035675 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.036365 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.036486 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:17.040927 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:17.040935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:17.040942 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:17.040989 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:17.044560 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:17.045643 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:17.045647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:17.045649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:17.045819 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:17.046442 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:17.046447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:17.046451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:17.046770 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:17.046945 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:17.048124 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0924 15:59:17.048417 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":246,"threads_started":1}
I20260924 15:59:17.052091 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:17.027263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41823 (local address 127.16.233.129:45209)
0924 15:59:17.028077 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:17.028118 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:17.029928 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:17.030683 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:17.030692 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:17.031223 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:17.031831 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:17.031855 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.033046 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.033051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:17.035509 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:17.035523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.036612 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.036625 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:17.036875 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:17.037752 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:17.037774 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:17.040574 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:17.044762 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:17.044773 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:17.044796 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:17.045471 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:17.045982 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:17.045988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:17.045994 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:17.046266 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:17.046941 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:17.046953 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:17.047563 (+ 610us) client_negotiation.cc:769] Sending connection context
0924 15:59:17.049186 (+ 1623us) client_negotiation.cc:240] Negotiation successful
0924 15:59:17.049953 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":369,"threads_started":1}
I20260924 15:59:17.073606 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:17.059216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:33995 (local address 127.16.233.190:41823)
0924 15:59:17.059471 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:17.059481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:17.059553 (+ 72us) server_negotiation.cc:407] Connection header received
0924 15:59:17.059759 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:17.059767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:17.059873 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:17.060007 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:17.060941 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.062458 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.064354 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.065134 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.065295 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:17.067667 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:17.067676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:17.067683 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:17.067729 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:17.070933 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:17.071625 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:17.071632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:17.071636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:17.071770 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:17.072134 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:17.072140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:17.072143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:17.072432 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:17.072590 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:17.073001 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 15:59:17.073228 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 15:59:17.073836 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:17.058802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41823 (local address 127.16.233.129:33995)
0924 15:59:17.059461 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:17.059481 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:17.059627 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:17.060039 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:17.060042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:17.060493 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:17.060791 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:17.060800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.062625 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.062630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:17.064121 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:17.064135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.065378 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.065386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:17.065633 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:17.066359 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:17.066387 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:17.067405 (+ 1018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:17.071074 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:17.071079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:17.071087 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:17.071501 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:17.071885 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:17.071891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:17.071895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:17.072027 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:17.072550 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:17.072555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:17.072758 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 15:59:17.072903 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 15:59:17.073245 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":482,"threads_started":1}
I20260924 15:59:17.078462 18526 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:34341
I20260924 15:59:17.078516 18662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:34341 every 8 connection(s)
I20260924 15:59:17.079622 18526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:17.081462 18663 heartbeater.cc:347] Connected to a master server at 127.16.233.190:41823
I20260924 15:59:17.081698 18663 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:17.082365 18663 heartbeater.cc:511] Master 127.16.233.190:41823 requested a full tablet report, sending...
I20260924 15:59:17.084754 18454 ts_manager.cc:194] Registered new tserver with Master: 0292be8689d04d4ab112eafa591b6d70 (127.16.233.129:34341)
I20260924 15:59:17.086140 18454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:45209
I20260924 15:59:17.087193 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 18526
I20260924 15:59:17.087430 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:17.088044 17318 external_mini_cluster.cc:1428] Setting key 54c83634e6a09fa8f912a01d3b495e93
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 15:59:17.129817 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41823
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:17.446849 18667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:17.447134 18667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:17.447230 18667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:17.454474 18667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:17.454607 18667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:17.454720 18667 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:59:17.454841 18667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:17.464758 18667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.190:41823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:17.466060 18667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:17.467447 18667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:17.482169 18673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:17.482141 18672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:17.483179 18675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:17.483748 18667 server_base.cc:1034] running on GCE node
I20260924 15:59:17.484483 18667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:17.485519 18667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:17.486764 18667 hybrid_clock.cc:648] HybridClock initialized: now 1790265557486689 us; error 51 us; skew 500 ppm
I20260924 15:59:17.487082 18667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:17 dist-test-slave-xn4h krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265557, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:17.490196 18667 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 15:59:17.491755 18667 webserver.cc:460] Webserver started at http://127.16.233.130:34777/ using document root <none> and password file <none>
I20260924 15:59:17.492481 18667 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:17.492597 18667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:17.492959 18667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:17.495959 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "4e5013313b77418e974d64a424969dea"
format_stamp: "Formatted at 2026-09-24 15:59:17 on dist-test-slave-xn4h"
server_key: "8b99040b40d65795924c9f585fda66d6"
server_key_iv: "c3047179fafcb1fe72793121790f46e4"
server_key_version: "encryptionkey@0"
I20260924 15:59:17.496760 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "4e5013313b77418e974d64a424969dea"
format_stamp: "Formatted at 2026-09-24 15:59:17 on dist-test-slave-xn4h"
server_key: "8b99040b40d65795924c9f585fda66d6"
server_key_iv: "c3047179fafcb1fe72793121790f46e4"
server_key_version: "encryptionkey@0"
I20260924 15:59:17.501983 18667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 15:59:17.505661 18682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:17.507045 18667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 15:59:17.507206 18667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "4e5013313b77418e974d64a424969dea"
format_stamp: "Formatted at 2026-09-24 15:59:17 on dist-test-slave-xn4h"
server_key: "8b99040b40d65795924c9f585fda66d6"
server_key_iv: "c3047179fafcb1fe72793121790f46e4"
server_key_version: "encryptionkey@0"
I20260924 15:59:17.507426 18667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:17.522943 18667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:17.527390 18667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:17.527666 18667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:17.530258 18667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:17.530400 18667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 15:59:17.530534 18667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:17.530604 18667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:59:17 dist-test-slave-xn4h krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265557, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:17.557006 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:17.530944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:36607 (local address 127.16.233.190:41823)
0924 15:59:17.531141 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:17.531149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:17.532621 (+ 1472us) server_negotiation.cc:407] Connection header received
0924 15:59:17.535003 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:17.535010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:17.535116 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:17.535262 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:17.538275 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.539299 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.542662 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.543289 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.543455 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:17.548700 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:17.548708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:17.548715 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:17.548774 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:17.552654 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:17.553779 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:17.553789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:17.553793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:17.553961 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:17.554607 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:17.554612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:17.554616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:17.555032 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:17.555210 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:17.556498 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0924 15:59:17.556744 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 15:59:17.559227 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:17.531678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41823 (local address 127.16.233.130:36607)
0924 15:59:17.532460 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:17.532501 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:17.534580 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:17.535895 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:17.535909 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:17.536941 (+ 1032us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:17.538021 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:17.538053 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.539502 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.539509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:17.542440 (+ 2931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:17.542467 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.543581 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.543590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:17.543853 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:17.544773 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:17.544823 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:17.548425 (+ 3602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:17.552859 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:17.552871 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:17.552903 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:17.553584 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:17.554105 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:17.554115 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:17.554119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:17.554471 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:17.555220 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:17.555232 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:17.555946 (+ 714us) client_negotiation.cc:769] Sending connection context
0924 15:59:17.556375 (+ 429us) client_negotiation.cc:240] Negotiation successful
0924 15:59:17.557142 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":342,"threads_started":1}
I20260924 15:59:17.570403 18667 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:39011
I20260924 15:59:17.570428 18802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:39011 every 8 connection(s)
I20260924 15:59:17.571828 18667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:17.574115 18803 heartbeater.cc:347] Connected to a master server at 127.16.233.190:41823
I20260924 15:59:17.574379 18803 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:17.575181 18803 heartbeater.cc:511] Master 127.16.233.190:41823 requested a full tablet report, sending...
I20260924 15:59:17.576486 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 18667
I20260924 15:59:17.576695 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:17.576817 18454 ts_manager.cc:194] Registered new tserver with Master: 4e5013313b77418e974d64a424969dea (127.16.233.130:39011)
I20260924 15:59:17.577281 17318 external_mini_cluster.cc:1428] Setting key a1b32e216afc7dbfb866b57275f04cfc
I20260924 15:59:17.578019 18454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:36607
I20260924 15:59:17.582715 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:17.567526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:50319 (local address 127.16.233.190:41823)
0924 15:59:17.567802 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:17.567812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:17.568721 (+ 909us) server_negotiation.cc:407] Connection header received
0924 15:59:17.568844 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:17.568850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:17.568938 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:17.569068 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:17.570800 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.571543 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.573603 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.574269 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.574442 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:17.576401 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:17.576407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:17.576411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:17.576452 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:17.579983 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:17.580683 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:17.580690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:17.580694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:17.580849 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:17.581267 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:17.581272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:17.581275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:17.581572 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:17.581663 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:17.582143 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 15:59:17.582392 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 15:59:17.582825 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:17.567853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41823 (local address 127.16.233.130:50319)
0924 15:59:17.568597 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:17.568616 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:17.568744 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:17.569270 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:17.569278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:17.570055 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:17.570642 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:17.570659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.571688 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.571696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:17.573473 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:17.573488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:17.574490 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:17.574500 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:17.574674 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:17.575388 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:17.575405 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:17.576219 (+ 814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:17.580120 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:17.580126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:17.580130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:17.580564 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:17.580962 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:17.580967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:17.580972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:17.581104 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:17.581743 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:17.581746 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:17.581886 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 15:59:17.582034 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 15:59:17.582392 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":430,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 15:59:17.620121 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41823
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:17.946201 18807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:17.946560 18807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:17.946666 18807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:17.954046 18807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:17.954201 18807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:17.954383 18807 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:59:17.954483 18807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:17.965592 18807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.190:41823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:17.967062 18807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:17.968444 18807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:17.982355 18812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:17.982462 18813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:17.982411 18816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:17.983248 18807 server_base.cc:1034] running on GCE node
I20260924 15:59:17.983916 18807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:17.984982 18807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:17.986284 18807 hybrid_clock.cc:648] HybridClock initialized: now 1790265557986216 us; error 46 us; skew 500 ppm
I20260924 15:59:17.986703 18807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:17 dist-test-slave-xn4h krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265557, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:17.989965 18807 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 15:59:17.991647 18807 webserver.cc:460] Webserver started at http://127.16.233.131:33557/ using document root <none> and password file <none>
I20260924 15:59:17.992441 18807 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:17.992537 18807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:17.992832 18807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:17.996009 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "0aae1d12570a43868fb1b8b8344807ba"
format_stamp: "Formatted at 2026-09-24 15:59:17 on dist-test-slave-xn4h"
server_key: "bfb7c98bd52bc0cee5302f8f17887172"
server_key_iv: "96047fc4cd50800cdff12abd201dbccb"
server_key_version: "encryptionkey@0"
I20260924 15:59:17.996831 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "0aae1d12570a43868fb1b8b8344807ba"
format_stamp: "Formatted at 2026-09-24 15:59:17 on dist-test-slave-xn4h"
server_key: "bfb7c98bd52bc0cee5302f8f17887172"
server_key_iv: "96047fc4cd50800cdff12abd201dbccb"
server_key_version: "encryptionkey@0"
I20260924 15:59:18.002285 18807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 15:59:18.006294 18823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:18.007851 18807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 15:59:18.008044 18807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "0aae1d12570a43868fb1b8b8344807ba"
format_stamp: "Formatted at 2026-09-24 15:59:17 on dist-test-slave-xn4h"
server_key: "bfb7c98bd52bc0cee5302f8f17887172"
server_key_iv: "96047fc4cd50800cdff12abd201dbccb"
server_key_version: "encryptionkey@0"
I20260924 15:59:18.008275 18807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:18.032099 18807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:18.036300 18807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:18.036605 18807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:18.039690 18807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:18.039798 18807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:18.039958 18807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:18.040102 18807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 15:59:18 dist-test-slave-xn4h krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265557, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:18.064273 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.040140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:51209 (local address 127.16.233.190:41823)
0924 15:59:18.040386 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:18.040396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:18.041918 (+ 1522us) server_negotiation.cc:407] Connection header received
0924 15:59:18.043998 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:18.044004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:18.044100 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:18.044217 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:18.046540 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.047637 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.050481 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.050934 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.051082 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.056865 (+ 5783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:18.056871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:18.056875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:18.056915 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:18.060469 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:18.061623 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:18.061629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:18.061633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:18.061789 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:18.062337 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:18.062347 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:18.062351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:18.062653 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:18.062815 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:18.063637 (+ 822us) server_negotiation.cc:299] Negotiation successful
0924 15:59:18.063876 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 15:59:18.065333 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.040929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41823 (local address 127.16.233.131:51209)
0924 15:59:18.041725 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:18.041773 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:18.043685 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:18.044471 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:18.044481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:18.045301 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:18.046276 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.046342 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.047896 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.047904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.050217 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.050232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.051250 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.051261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.051652 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.052550 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:18.052592 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:18.056504 (+ 3912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:18.060636 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:18.060647 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:18.060667 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:18.061433 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:18.061917 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:18.061922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:18.061926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:18.062153 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:18.062798 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:18.062806 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:18.063196 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 15:59:18.063474 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 15:59:18.064068 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":329,"threads_started":1}
I20260924 15:59:18.088945 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.071613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:41961 (local address 127.16.233.190:41823)
0924 15:59:18.071833 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:18.071842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:18.072478 (+ 636us) server_negotiation.cc:407] Connection header received
0924 15:59:18.072643 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:18.072651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:18.072772 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:18.072914 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:18.073932 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.075457 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.076867 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.077754 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.077894 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.080637 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:18.080644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:18.080648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:18.080692 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:18.085120 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:18.086466 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:18.086498 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:18.086504 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:18.086688 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:18.087160 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:18.087168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:18.087174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:18.087588 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:18.087762 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:18.088340 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 15:59:18.088584 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 15:59:18.089077 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.071578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41823 (local address 127.16.233.131:41961)
0924 15:59:18.072345 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:18.072372 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:18.072509 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:18.072970 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:18.072974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:18.073441 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:18.073755 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.073777 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.075654 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.075659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.076704 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.076717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.078000 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.078009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.078211 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.079181 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:18.079210 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:18.080328 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:18.085361 (+ 5033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:18.085372 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:18.085379 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:18.086157 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:18.086833 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:18.086841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:18.086848 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:18.087014 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:18.087737 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:18.087754 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:18.088025 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 15:59:18.088214 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 15:59:18.088608 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":461,"threads_started":1}
I20260924 15:59:18.089108 18663 heartbeater.cc:503] Master 127.16.233.190:41823 was elected leader, sending a full tablet report...
I20260924 15:59:18.103592 18807 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:32795
I20260924 15:59:18.103621 18943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:32795 every 8 connection(s)
I20260924 15:59:18.104909 18807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:18.107100 18944 heartbeater.cc:347] Connected to a master server at 127.16.233.190:41823
I20260924 15:59:18.107381 18944 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:18.108049 18944 heartbeater.cc:511] Master 127.16.233.190:41823 requested a full tablet report, sending...
I20260924 15:59:18.109542 18453 ts_manager.cc:194] Registered new tserver with Master: 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131:32795)
I20260924 15:59:18.110028 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 18807
I20260924 15:59:18.110232 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:18.110601 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:51209
I20260924 15:59:18.110955 17318 external_mini_cluster.cc:1428] Setting key 959de3a1ff01eae4cf1a05a53da25b58
I20260924 15:59:18.114462 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:59:18 dist-test-slave-xn4h krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:59:18 dist-test-slave-xn4h krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:18.168411 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.153339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.16.233.190:41823)
0924 15:59:18.153595 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:18.153603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:18.153862 (+ 259us) server_negotiation.cc:407] Connection header received
0924 15:59:18.154085 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:18.154094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:18.154159 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:18.154276 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:18.155867 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.156667 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.157991 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.158492 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.158637 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.163015 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:18.163021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:18.163025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:18.163066 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:18.165564 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:18.166296 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:18.166319 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:18.166325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:18.166475 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:18.166878 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:18.166885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:18.166893 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:18.167200 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:18.167473 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:18.167821 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 15:59:18.168061 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 15:59:18.172755 18453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:59:18.179751 18453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:18.206372 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.192297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39011 (local address 127.0.0.1:37566)
0924 15:59:18.193851 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:18.193891 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:18.194105 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:18.196471 (+ 2366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:18.196479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:18.196515 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:18.197042 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.197057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.200876 (+ 3819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.200883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.203093 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.203109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.205233 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.205240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.205444 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.205501 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 15:59:18.205655 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 15:59:18.205887 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"mutex_wait_us":1,"thread_start_us":1206,"threads_started":1}
I20260924 15:59:18.206508 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.192840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.16.233.129:34341)
0924 15:59:18.193490 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:18.193501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:18.194089 (+ 588us) server_negotiation.cc:407] Connection header received
0924 15:59:18.194425 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:18.194431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:18.194674 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:18.194860 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:18.196114 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.198209 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.203488 (+ 5279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.205092 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.205280 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.205453 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:18.206004 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 15:59:18.206183 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":322,"threads_started":1}
I20260924 15:59:18.206573 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.192328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.16.233.130:39011)
0924 15:59:18.193359 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:18.193370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:18.194005 (+ 635us) server_negotiation.cc:407] Connection header received
0924 15:59:18.194352 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:18.194361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:18.194660 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:18.194850 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:18.197265 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.200650 (+ 3385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.203620 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.205022 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.205209 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.205406 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:18.205699 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 15:59:18.205886 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":689,"threads_started":1}
I20260924 15:59:18.207645 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.192360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:34341 (local address 127.0.0.1:36808)
0924 15:59:18.193978 (+ 1618us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:18.193998 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:18.194127 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:18.195206 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:18.195218 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:18.195281 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:18.195847 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.195863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.198505 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.198513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.203334 (+ 4821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.203342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.205599 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.205605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.205793 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.205833 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 15:59:18.206960 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0924 15:59:18.207121 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1475,"mutex_wait_us":1476,"thread_start_us":1808,"threads_started":1}
I20260924 15:59:18.211844 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.194735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:32795 (local address 127.0.0.1:37006)
0924 15:59:18.202987 (+ 8252us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:18.203004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:18.203116 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:18.204123 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:18.204128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:18.204154 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:18.204521 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.204533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.206658 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.206663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.209067 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.209081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.210933 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.210937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.211086 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.211120 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 15:59:18.211217 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 15:59:18.211319 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8116,"mutex_wait_us":1,"thread_start_us":7966,"threads_started":1}
I20260924 15:59:18.212064 18732 tablet_service.cc:1467] Processing CreateTablet for tablet ca1b2d5a613e41f39c2964326d3113fe (DEFAULT_TABLE table=test-table [id=29a8f27f385440ffbcfa576687d52d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:18.212064 18593 tablet_service.cc:1467] Processing CreateTablet for tablet ca1b2d5a613e41f39c2964326d3113fe (DEFAULT_TABLE table=test-table [id=29a8f27f385440ffbcfa576687d52d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:18.213001 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.195132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.16.233.131:32795)
0924 15:59:18.202489 (+ 7357us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:18.202504 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:18.203116 (+ 612us) server_negotiation.cc:407] Connection header received
0924 15:59:18.203236 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:18.203247 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:18.203566 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:18.203779 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:18.204756 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.206482 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.209268 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.210510 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.210706 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.210825 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:18.212201 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0924 15:59:18.212438 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7061,"thread_start_us":6835,"threads_started":1}
I20260924 15:59:18.213778 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1b2d5a613e41f39c2964326d3113fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:18.213795 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1b2d5a613e41f39c2964326d3113fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:18.217135 18873 tablet_service.cc:1467] Processing CreateTablet for tablet ca1b2d5a613e41f39c2964326d3113fe (DEFAULT_TABLE table=test-table [id=29a8f27f385440ffbcfa576687d52d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:18.218406 18873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1b2d5a613e41f39c2964326d3113fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:18.233817 18964 tablet_bootstrap.cc:492] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: Bootstrap starting.
I20260924 15:59:18.235548 18966 tablet_bootstrap.cc:492] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: Bootstrap starting.
I20260924 15:59:18.238745 18965 tablet_bootstrap.cc:492] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: Bootstrap starting.
I20260924 15:59:18.238780 18966 tablet_bootstrap.cc:654] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:18.240126 18966 log.cc:824] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:18.242276 18965 tablet_bootstrap.cc:654] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:18.242343 18964 tablet_bootstrap.cc:654] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:18.243405 18965 log.cc:824] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:18.243599 18964 log.cc:824] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:18.245152 18966 tablet_bootstrap.cc:492] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: No bootstrap required, opened a new log
I20260924 15:59:18.245450 18966 ts_tablet_manager.cc:1397] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260924 15:59:18.246637 18964 tablet_bootstrap.cc:492] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: No bootstrap required, opened a new log
I20260924 15:59:18.246817 18965 tablet_bootstrap.cc:492] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: No bootstrap required, opened a new log
I20260924 15:59:18.246960 18964 ts_tablet_manager.cc:1397] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20260924 15:59:18.247189 18965 ts_tablet_manager.cc:1397] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 15:59:18.254801 18966 raft_consensus.cc:348] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.255264 18966 raft_consensus.cc:374] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:18.255362 18966 raft_consensus.cc:729] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aae1d12570a43868fb1b8b8344807ba, State: Initialized, Role: FOLLOWER
I20260924 15:59:18.256299 18966 consensus_queue.cc:260] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.259117 18944 heartbeater.cc:503] Master 127.16.233.190:41823 was elected leader, sending a full tablet report...
I20260924 15:59:18.260037 18966 ts_tablet_manager.cc:1428] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260924 15:59:18.260138 18964 raft_consensus.cc:348] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.260239 18965 raft_consensus.cc:348] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.260967 18964 raft_consensus.cc:374] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:18.261075 18965 raft_consensus.cc:374] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:18.261171 18964 raft_consensus.cc:729] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5013313b77418e974d64a424969dea, State: Initialized, Role: FOLLOWER
I20260924 15:59:18.261262 18965 raft_consensus.cc:729] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0292be8689d04d4ab112eafa591b6d70, State: Initialized, Role: FOLLOWER
I20260924 15:59:18.261937 18965 consensus_queue.cc:260] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.262038 18964 consensus_queue.cc:260] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.264184 18965 ts_tablet_manager.cc:1428] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260924 15:59:18.265107 18803 heartbeater.cc:503] Master 127.16.233.190:41823 was elected leader, sending a full tablet report...
I20260924 15:59:18.265765 18964 ts_tablet_manager.cc:1428] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
W20260924 15:59:18.324056 18804 tablet.cc:2369] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:59:18.331948 18664 tablet.cc:2369] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:59:18.356566 18945 tablet.cc:2369] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:59:18.537160 18970 raft_consensus.cc:482] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:18.537626 18970 raft_consensus.cc:504] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.539861 18970 leader_election.cc:288] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e5013313b77418e974d64a424969dea (127.16.233.130:39011), 0292be8689d04d4ab112eafa591b6d70 (127.16.233.129:34341)
I20260924 15:59:18.545972 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.540320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:34341 (local address 127.16.233.131:55143)
0924 15:59:18.540511 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:18.540533 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:18.540683 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:18.541026 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:18.541031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:18.541060 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:18.541280 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.541289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.542328 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.542336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.543981 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.543994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.545165 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.545188 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.545384 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.545409 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 15:59:18.545514 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 15:59:18.545650 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 15:59:18.545995 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.540303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:55143 (local address 127.16.233.129:34341)
0924 15:59:18.540512 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:18.540519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:18.540615 (+ 96us) server_negotiation.cc:407] Connection header received
0924 15:59:18.540843 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:18.540847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:18.540907 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:18.541005 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:18.541428 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.542133 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.544149 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.545000 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.545205 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.545301 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:18.545540 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 15:59:18.545669 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 15:59:18.547339 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.540289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:43169 (local address 127.16.233.130:39011)
0924 15:59:18.540510 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:18.540517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:18.541253 (+ 736us) server_negotiation.cc:407] Connection header received
0924 15:59:18.541469 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:18.541479 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:18.541536 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:18.541630 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:18.542704 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.543416 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.545405 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.546338 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.546503 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.546553 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:18.546953 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 15:59:18.547090 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 15:59:18.547410 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.540320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39011 (local address 127.16.233.131:43169)
0924 15:59:18.541143 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:18.541158 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:18.541318 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:18.541643 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:18.541648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:18.541672 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:18.542493 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.542504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.543601 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.543604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.545235 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:18.545247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.546583 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.546590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:18.546778 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.546816 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:59:18.546891 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:59:18.546979 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":162,"thread_start_us":349,"threads_started":1}
I20260924 15:59:18.547307 18613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1b2d5a613e41f39c2964326d3113fe" candidate_uuid: "0aae1d12570a43868fb1b8b8344807ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0292be8689d04d4ab112eafa591b6d70" is_pre_election: true
I20260924 15:59:18.547941 18613 raft_consensus.cc:2454] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aae1d12570a43868fb1b8b8344807ba in term 0.
I20260924 15:59:18.548626 18827 leader_election.cc:302] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0292be8689d04d4ab112eafa591b6d70, 0aae1d12570a43868fb1b8b8344807ba; no voters:
I20260924 15:59:18.549015 18970 raft_consensus.cc:2781] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:18.549140 18970 raft_consensus.cc:482] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:18.548956 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1b2d5a613e41f39c2964326d3113fe" candidate_uuid: "0aae1d12570a43868fb1b8b8344807ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5013313b77418e974d64a424969dea" is_pre_election: true
I20260924 15:59:18.549260 18970 raft_consensus.cc:3037] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:18.549577 18752 raft_consensus.cc:2454] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aae1d12570a43868fb1b8b8344807ba in term 0.
I20260924 15:59:18.552670 18970 raft_consensus.cc:504] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.553401 18970 leader_election.cc:288] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [CANDIDATE]: Term 1 election: Requested vote from peers 4e5013313b77418e974d64a424969dea (127.16.233.130:39011), 0292be8689d04d4ab112eafa591b6d70 (127.16.233.129:34341)
I20260924 15:59:18.554013 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1b2d5a613e41f39c2964326d3113fe" candidate_uuid: "0aae1d12570a43868fb1b8b8344807ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5013313b77418e974d64a424969dea"
I20260924 15:59:18.554040 18613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1b2d5a613e41f39c2964326d3113fe" candidate_uuid: "0aae1d12570a43868fb1b8b8344807ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0292be8689d04d4ab112eafa591b6d70"
I20260924 15:59:18.554260 18613 raft_consensus.cc:3037] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:18.554252 18752 raft_consensus.cc:3037] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:18.558280 18613 raft_consensus.cc:2454] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aae1d12570a43868fb1b8b8344807ba in term 1.
I20260924 15:59:18.558280 18752 raft_consensus.cc:2454] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aae1d12570a43868fb1b8b8344807ba in term 1.
I20260924 15:59:18.558852 18826 leader_election.cc:302] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aae1d12570a43868fb1b8b8344807ba, 4e5013313b77418e974d64a424969dea; no voters:
I20260924 15:59:18.559326 18970 raft_consensus.cc:2781] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:18.559867 18970 raft_consensus.cc:686] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 1 LEADER]: Becoming Leader. State: Replica: 0aae1d12570a43868fb1b8b8344807ba, State: Running, Role: LEADER
I20260924 15:59:18.560400 18970 consensus_queue.cc:237] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.565349 18454 catalog_manager.cc:5495] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131). New cstate: current_term: 1 leader_uuid: "0aae1d12570a43868fb1b8b8344807ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } health_report { overall_health: HEALTHY } } }
I20260924 15:59:18.674844 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:18.669001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.16.233.131:32795)
0924 15:59:18.669241 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:18.669251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:18.669651 (+ 400us) server_negotiation.cc:407] Connection header received
0924 15:59:18.669961 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:18.669965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:18.670042 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:18.670163 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:18.670657 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.671344 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.672662 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:18.673067 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:18.673157 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:18.673614 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:18.674144 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:18.674255 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:18.674455 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 15:59:18.674528 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 15:59:18.683911 18613 raft_consensus.cc:1261] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Refusing update from remote peer 0aae1d12570a43868fb1b8b8344807ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:18.683911 18752 raft_consensus.cc:1261] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [term 1 FOLLOWER]: Refusing update from remote peer 0aae1d12570a43868fb1b8b8344807ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:18.684932 18974 consensus_queue.cc:1035] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:18.685315 18970 consensus_queue.cc:1035] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:18.725124 18454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:18.725411 18454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:18.729084 18454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe: Sending DeleteTablet for 3 replicas of tablet ca1b2d5a613e41f39c2964326d3113fe
I20260924 15:59:18.730487 18732 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1b2d5a613e41f39c2964326d3113fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:18 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:37566
I20260924 15:59:18.730542 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1b2d5a613e41f39c2964326d3113fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:18 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:36808
I20260924 15:59:18.731303 18873 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1b2d5a613e41f39c2964326d3113fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:18 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:37006
I20260924 15:59:18.734696 18454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:59:18.735452 18454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:18.748513 18989 tablet_replica.cc:330] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: stopping tablet replica
I20260924 15:59:18.749202 18872 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee093bf37a84fa4b464744a9e5fc1ec (DEFAULT_TABLE table=test-table [id=10e2b92516494c2581b5505529a6fbe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:18.749317 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee093bf37a84fa4b464744a9e5fc1ec (DEFAULT_TABLE table=test-table [id=10e2b92516494c2581b5505529a6fbe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:18.749922 18872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee093bf37a84fa4b464744a9e5fc1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:18.749929 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee093bf37a84fa4b464744a9e5fc1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:18.750075 18989 raft_consensus.cc:2229] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:18.750953 18989 raft_consensus.cc:2258] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:18.752128 18988 tablet_replica.cc:330] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: stopping tablet replica
I20260924 15:59:18.752745 18988 raft_consensus.cc:2229] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:18.753289 18988 raft_consensus.cc:2258] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:18.753633 18989 ts_tablet_manager.cc:1905] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:18.754693 18731 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee093bf37a84fa4b464744a9e5fc1ec (DEFAULT_TABLE table=test-table [id=10e2b92516494c2581b5505529a6fbe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:18.755239 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee093bf37a84fa4b464744a9e5fc1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:18.756752 18965 tablet_bootstrap.cc:492] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: Bootstrap starting.
I20260924 15:59:18.758754 18965 tablet_bootstrap.cc:654] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:18.762720 18965 tablet_bootstrap.cc:492] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: No bootstrap required, opened a new log
I20260924 15:59:18.762955 18988 ts_tablet_manager.cc:1905] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:18.763733 18965 ts_tablet_manager.cc:1397] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 15:59:18.764833 18965 raft_consensus.cc:348] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.765259 18965 raft_consensus.cc:374] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:18.765548 18965 raft_consensus.cc:729] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0292be8689d04d4ab112eafa591b6d70, State: Initialized, Role: FOLLOWER
I20260924 15:59:18.765708 18964 tablet_bootstrap.cc:492] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: Bootstrap starting.
I20260924 15:59:18.766196 18965 consensus_queue.cc:260] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.768483 18965 ts_tablet_manager.cc:1428] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 15:59:18.769837 18964 tablet_bootstrap.cc:654] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:18.775552 18989 ts_tablet_manager.cc:1918] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:59:18.775772 18989 log.cc:1196] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/ca1b2d5a613e41f39c2964326d3113fe
I20260924 15:59:18.776379 18989 ts_tablet_manager.cc:1939] T ca1b2d5a613e41f39c2964326d3113fe P 4e5013313b77418e974d64a424969dea: Deleting consensus metadata
I20260924 15:59:18.776585 18990 tablet_replica.cc:330] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: stopping tablet replica
I20260924 15:59:18.777122 18990 raft_consensus.cc:2229] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:59:18.779733 18988 ts_tablet_manager.cc:1918] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:59:18.779973 18988 log.cc:1196] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/ca1b2d5a613e41f39c2964326d3113fe
I20260924 15:59:18.778676 18440 catalog_manager.cc:4841] TS 4e5013313b77418e974d64a424969dea (127.16.233.130:39011): tablet ca1b2d5a613e41f39c2964326d3113fe (table test-table [id=29a8f27f385440ffbcfa576687d52d53]) successfully deleted
I20260924 15:59:18.783928 18988 ts_tablet_manager.cc:1939] T ca1b2d5a613e41f39c2964326d3113fe P 0292be8689d04d4ab112eafa591b6d70: Deleting consensus metadata
I20260924 15:59:18.784417 18964 tablet_bootstrap.cc:492] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: No bootstrap required, opened a new log
I20260924 15:59:18.784593 18964 ts_tablet_manager.cc:1397] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.006s
I20260924 15:59:18.786087 18964 raft_consensus.cc:348] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.786468 18964 raft_consensus.cc:374] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:18.786646 18964 raft_consensus.cc:729] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5013313b77418e974d64a424969dea, State: Initialized, Role: FOLLOWER
I20260924 15:59:18.786685 18441 catalog_manager.cc:4841] TS 0292be8689d04d4ab112eafa591b6d70 (127.16.233.129:34341): tablet ca1b2d5a613e41f39c2964326d3113fe (table test-table [id=29a8f27f385440ffbcfa576687d52d53]) successfully deleted
I20260924 15:59:18.786553 18990 raft_consensus.cc:2258] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:18.786970 18964 consensus_queue.cc:260] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.787762 18993 tablet_bootstrap.cc:492] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: Bootstrap starting.
I20260924 15:59:18.788391 18964 ts_tablet_manager.cc:1428] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 15:59:18.789376 18990 ts_tablet_manager.cc:1905] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:18.789817 18993 tablet_bootstrap.cc:654] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:18.797354 18993 tablet_bootstrap.cc:492] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: No bootstrap required, opened a new log
I20260924 15:59:18.797560 18993 ts_tablet_manager.cc:1397] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 15:59:18.797684 18990 ts_tablet_manager.cc:1918] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:59:18.798069 18990 log.cc:1196] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/ca1b2d5a613e41f39c2964326d3113fe
I20260924 15:59:18.798514 18993 raft_consensus.cc:348] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.798808 18993 raft_consensus.cc:374] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:18.798943 18990 ts_tablet_manager.cc:1939] T ca1b2d5a613e41f39c2964326d3113fe P 0aae1d12570a43868fb1b8b8344807ba: Deleting consensus metadata
I20260924 15:59:18.798957 18993 raft_consensus.cc:729] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aae1d12570a43868fb1b8b8344807ba, State: Initialized, Role: FOLLOWER
I20260924 15:59:18.799360 18993 consensus_queue.cc:260] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:18.800757 18993 ts_tablet_manager.cc:1428] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:18.801748 18441 catalog_manager.cc:4841] TS 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131:32795): tablet ca1b2d5a613e41f39c2964326d3113fe (table test-table [id=29a8f27f385440ffbcfa576687d52d53]) successfully deleted
I20260924 15:59:19.085251 18972 raft_consensus.cc:482] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:19.085475 18972 raft_consensus.cc:504] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.087448 18972 leader_election.cc:288] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0292be8689d04d4ab112eafa591b6d70 (127.16.233.129:34341), 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131:32795)
I20260924 15:59:19.088047 18971 raft_consensus.cc:482] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:19.088229 18971 raft_consensus.cc:504] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.089643 18971 leader_election.cc:288] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e5013313b77418e974d64a424969dea (127.16.233.130:39011), 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131:32795)
I20260924 15:59:19.103942 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.087658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:34341 (local address 127.16.233.130:60193)
0924 15:59:19.088160 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:19.088182 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:19.088324 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:19.094144 (+ 5820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:19.094152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:19.094180 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:19.094623 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.094639 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.096835 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.096841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.100196 (+ 3355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.100219 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.102790 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.102799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.103092 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.103156 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 15:59:19.103243 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 15:59:19.103468 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":2774,"threads_started":1}
I20260924 15:59:19.105808 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.090041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:60193 (local address 127.16.233.129:34341)
0924 15:59:19.093791 (+ 3750us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.093799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.093824 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:59:19.093896 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.093900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.093976 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.094834 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:19.095149 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.096640 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.100502 (+ 3862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.102538 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.104895 (+ 2357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.105016 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.105081 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.105287 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3600,"spinlock_wait_cycles":28544,"thread_start_us":3487,"threads_started":1}
I20260924 15:59:19.114130 18613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee093bf37a84fa4b464744a9e5fc1ec" candidate_uuid: "4e5013313b77418e974d64a424969dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0292be8689d04d4ab112eafa591b6d70" is_pre_election: true
I20260924 15:59:19.114449 18613 raft_consensus.cc:2454] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e5013313b77418e974d64a424969dea in term 0.
I20260924 15:59:19.115167 18686 leader_election.cc:302] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0292be8689d04d4ab112eafa591b6d70, 4e5013313b77418e974d64a424969dea; no voters:
I20260924 15:59:19.115658 18972 raft_consensus.cc:2781] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:19.115779 18972 raft_consensus.cc:482] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:19.115890 18972 raft_consensus.cc:3037] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:19.124553 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.090417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:56683 (local address 127.16.233.131:32795)
0924 15:59:19.090606 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.090611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.113818 (+ 23207us) server_negotiation.cc:407] Connection header received
0924 15:59:19.114030 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.114036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.114118 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.114224 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:19.115079 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.116272 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.121637 (+ 5365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.123083 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.123220 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.123313 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.124083 (+ 770us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.124232 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 15:59:19.125671 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.090042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:32795 (local address 127.16.233.129:56683)
0924 15:59:19.113681 (+ 23639us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:19.113707 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:19.113899 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:19.114440 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:19.114445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:19.114477 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:19.114886 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.114900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.116507 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.116513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.121290 (+ 4777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.121312 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.123592 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.123600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.123793 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.123827 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 15:59:19.124839 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0924 15:59:19.125006 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23479,"threads_started":1}
I20260924 15:59:19.132247 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee093bf37a84fa4b464744a9e5fc1ec" candidate_uuid: "0292be8689d04d4ab112eafa591b6d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aae1d12570a43868fb1b8b8344807ba" is_pre_election: true
I20260924 15:59:19.132762 18899 raft_consensus.cc:2454] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0292be8689d04d4ab112eafa591b6d70 in term 0.
I20260924 15:59:19.133165 18972 raft_consensus.cc:504] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.133882 18972 leader_election.cc:288] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [CANDIDATE]: Term 1 election: Requested vote from peers 0292be8689d04d4ab112eafa591b6d70 (127.16.233.129:34341), 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131:32795)
I20260924 15:59:19.134323 18546 leader_election.cc:302] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0292be8689d04d4ab112eafa591b6d70, 0aae1d12570a43868fb1b8b8344807ba; no voters:
I20260924 15:59:19.135102 18971 raft_consensus.cc:2781] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:19.135231 18971 raft_consensus.cc:482] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:19.135335 18971 raft_consensus.cc:3037] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:19.136566 18613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee093bf37a84fa4b464744a9e5fc1ec" candidate_uuid: "4e5013313b77418e974d64a424969dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0292be8689d04d4ab112eafa591b6d70"
I20260924 15:59:19.137086 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.090962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:32795 (local address 127.16.233.130:60703)
0924 15:59:19.093093 (+ 2131us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:19.093118 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:19.093275 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:19.119489 (+ 26214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:19.119498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:19.119549 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:19.119966 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.119987 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.128509 (+ 8522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.128517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.132528 (+ 4011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.132542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.135830 (+ 3288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.135839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.136096 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.136134 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:59:19.136206 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:59:19.136359 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1972,"thread_start_us":1778,"threads_started":1}
I20260924 15:59:19.138599 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.091016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:60703 (local address 127.16.233.131:32795)
0924 15:59:19.118925 (+ 27909us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.118931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.118952 (+ 21us) server_negotiation.cc:407] Connection header received
0924 15:59:19.119044 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.119050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.119138 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.119307 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:19.126938 (+ 7631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.128238 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.134080 (+ 5842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.135466 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.137702 (+ 2236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.137825 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.137889 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.138032 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27757,"thread_start_us":30434,"threads_started":1}
I20260924 15:59:19.139308 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee093bf37a84fa4b464744a9e5fc1ec" candidate_uuid: "4e5013313b77418e974d64a424969dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aae1d12570a43868fb1b8b8344807ba" is_pre_election: true
I20260924 15:59:19.139576 18899 raft_consensus.cc:2454] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e5013313b77418e974d64a424969dea in term 0.
I20260924 15:59:19.139456 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee093bf37a84fa4b464744a9e5fc1ec" candidate_uuid: "4e5013313b77418e974d64a424969dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aae1d12570a43868fb1b8b8344807ba"
I20260924 15:59:19.139784 18898 raft_consensus.cc:3037] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:19.140751 18971 raft_consensus.cc:504] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.141978 18613 raft_consensus.cc:2379] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e5013313b77418e974d64a424969dea in current term 1: Already voted for candidate 0292be8689d04d4ab112eafa591b6d70 in this term.
I20260924 15:59:19.144652 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.094242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:55557 (local address 127.16.233.130:39011)
0924 15:59:19.099357 (+ 5115us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.099363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.126146 (+ 26783us) server_negotiation.cc:407] Connection header received
0924 15:59:19.126470 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.126476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.126598 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.126667 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:19.127614 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.131390 (+ 3776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.141722 (+ 10332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.143180 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.143362 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.143444 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.144079 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.144257 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4848,"thread_start_us":4603,"threads_started":1}
I20260924 15:59:19.145335 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.093909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39011 (local address 127.16.233.129:55557)
0924 15:59:19.126035 (+ 32126us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:19.126064 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:19.126255 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:19.127203 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:19.127208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:19.127239 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:19.127417 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.127430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.138837 (+ 11407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.138843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.141459 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.141482 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.143615 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.143624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.143855 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.143896 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 15:59:19.144861 (+ 965us) client_negotiation.cc:240] Negotiation successful
0924 15:59:19.145014 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31893,"thread_start_us":19391,"threads_started":1}
I20260924 15:59:19.146108 18971 leader_election.cc:288] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [CANDIDATE]: Term 1 election: Requested vote from peers 4e5013313b77418e974d64a424969dea (127.16.233.130:39011), 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131:32795)
I20260924 15:59:19.146257 18898 raft_consensus.cc:2454] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e5013313b77418e974d64a424969dea in term 1.
I20260924 15:59:19.146888 18686 leader_election.cc:302] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0aae1d12570a43868fb1b8b8344807ba, 4e5013313b77418e974d64a424969dea; no voters: 0292be8689d04d4ab112eafa591b6d70
I20260924 15:59:19.147358 18972 raft_consensus.cc:2781] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:19.149261 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee093bf37a84fa4b464744a9e5fc1ec" candidate_uuid: "0292be8689d04d4ab112eafa591b6d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aae1d12570a43868fb1b8b8344807ba"
I20260924 15:59:19.150318 18898 raft_consensus.cc:2379] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0292be8689d04d4ab112eafa591b6d70 in current term 1: Already voted for candidate 4e5013313b77418e974d64a424969dea in this term.
I20260924 15:59:19.153118 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee093bf37a84fa4b464744a9e5fc1ec" candidate_uuid: "0292be8689d04d4ab112eafa591b6d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5013313b77418e974d64a424969dea"
I20260924 15:59:19.153475 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee093bf37a84fa4b464744a9e5fc1ec" candidate_uuid: "0292be8689d04d4ab112eafa591b6d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5013313b77418e974d64a424969dea" is_pre_election: true
I20260924 15:59:19.154541 18972 raft_consensus.cc:686] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 1 LEADER]: Becoming Leader. State: Replica: 4e5013313b77418e974d64a424969dea, State: Running, Role: LEADER
I20260924 15:59:19.154975 18972 consensus_queue.cc:237] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.159543 18454 catalog_manager.cc:5495] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e5013313b77418e974d64a424969dea (127.16.233.130). New cstate: current_term: 1 leader_uuid: "4e5013313b77418e974d64a424969dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } health_report { overall_health: UNKNOWN } } }
I20260924 15:59:19.162153 18545 leader_election.cc:302] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0292be8689d04d4ab112eafa591b6d70; no voters: 0aae1d12570a43868fb1b8b8344807ba, 4e5013313b77418e974d64a424969dea
I20260924 15:59:19.163270 18971 raft_consensus.cc:2726] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
Sep 24 15:59:19 dist-test-slave-xn4h krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265556, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:19.293263 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.264531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41823 (local address 127.0.0.1:50062)
0924 15:59:19.265571 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:19.265604 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:19.265803 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:19.266151 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:19.266156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:19.266996 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:19.267305 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.267315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.268737 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.268748 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.269832 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.269844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.270602 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.270607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.270992 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.284011 (+ 13019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:19.284035 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:19.287047 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:19.290741 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:19.290757 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:19.290764 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:19.291352 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:19.291686 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:19.291689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:19.291691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:19.291855 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:19.292363 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:19.292370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:19.292542 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 15:59:19.292687 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 15:59:19.292915 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":535,"threads_started":1}
I20260924 15:59:19.293349 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.264548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.16.233.190:41823)
0924 15:59:19.265736 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.265742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.265789 (+ 47us) server_negotiation.cc:407] Connection header received
0924 15:59:19.265952 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.265957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.266020 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.266128 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:19.267528 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.268584 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.269977 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.270438 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.270576 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.287315 (+ 16739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:19.287321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:19.287325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:19.287361 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:19.290577 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.291467 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.291471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.291474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.291590 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.291954 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.291958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.291960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.292231 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:19.292339 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.292768 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.293038 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":976,"thread_start_us":700,"threads_started":1}
I20260924 15:59:19.298537 18454 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:50062:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:59:19.299613 18454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:19.306371 18592 tablet_service.cc:1467] Processing CreateTablet for tablet b446961fd5254e2980f081b3fb8f207e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=337a2d169b7e452a8963e0111fa77b49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:59:19.306522 18731 tablet_service.cc:1467] Processing CreateTablet for tablet b446961fd5254e2980f081b3fb8f207e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=337a2d169b7e452a8963e0111fa77b49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:59:19.306963 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b446961fd5254e2980f081b3fb8f207e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:19.306864 18872 tablet_service.cc:1467] Processing CreateTablet for tablet b446961fd5254e2980f081b3fb8f207e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=337a2d169b7e452a8963e0111fa77b49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:59:19.307057 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b446961fd5254e2980f081b3fb8f207e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:19.307364 18872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b446961fd5254e2980f081b3fb8f207e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:19.313853 19011 tablet_bootstrap.cc:492] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: Bootstrap starting.
I20260924 15:59:19.316594 19011 tablet_bootstrap.cc:654] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:19.319643 19013 tablet_bootstrap.cc:492] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: Bootstrap starting.
I20260924 15:59:19.320106 19011 tablet_bootstrap.cc:492] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: No bootstrap required, opened a new log
I20260924 15:59:19.320180 19012 tablet_bootstrap.cc:492] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: Bootstrap starting.
I20260924 15:59:19.320322 19011 ts_tablet_manager.cc:1397] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 15:59:19.321056 19011 raft_consensus.cc:348] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.321337 19011 raft_consensus.cc:374] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:19.321431 19011 raft_consensus.cc:729] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5013313b77418e974d64a424969dea, State: Initialized, Role: FOLLOWER
I20260924 15:59:19.321660 19011 consensus_queue.cc:260] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.322125 18972 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.322283 19011 ts_tablet_manager.cc:1428] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 15:59:19.322466 18972 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:19.323164 19012 tablet_bootstrap.cc:654] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:19.323721 19013 tablet_bootstrap.cc:654] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:19.326632 19013 tablet_bootstrap.cc:492] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: No bootstrap required, opened a new log
I20260924 15:59:19.326839 19013 ts_tablet_manager.cc:1397] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 15:59:19.327881 19013 raft_consensus.cc:348] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.328344 19013 raft_consensus.cc:374] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:19.328634 19013 raft_consensus.cc:729] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0292be8689d04d4ab112eafa591b6d70, State: Initialized, Role: FOLLOWER
I20260924 15:59:19.329000 19013 consensus_queue.cc:260] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.329581 18971 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.329862 19013 ts_tablet_manager.cc:1428] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260924 15:59:19.329912 18971 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:19.332170 19012 tablet_bootstrap.cc:492] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: No bootstrap required, opened a new log
I20260924 15:59:19.332413 19012 ts_tablet_manager.cc:1397] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260924 15:59:19.333676 19012 raft_consensus.cc:348] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.333941 19012 raft_consensus.cc:374] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:19.334051 19012 raft_consensus.cc:729] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aae1d12570a43868fb1b8b8344807ba, State: Initialized, Role: FOLLOWER
I20260924 15:59:19.334270 19012 consensus_queue.cc:260] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.335085 19012 ts_tablet_manager.cc:1428] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:59:19.334975 19017 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.335310 19017 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:19.341216 18971 raft_consensus.cc:482] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:19.341464 18971 raft_consensus.cc:504] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.342080 18971 leader_election.cc:288] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e5013313b77418e974d64a424969dea (127.16.233.130:39011), 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131:32795)
I20260924 15:59:19.342645 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b446961fd5254e2980f081b3fb8f207e" candidate_uuid: "0292be8689d04d4ab112eafa591b6d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aae1d12570a43868fb1b8b8344807ba" is_pre_election: true
I20260924 15:59:19.342645 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b446961fd5254e2980f081b3fb8f207e" candidate_uuid: "0292be8689d04d4ab112eafa591b6d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5013313b77418e974d64a424969dea" is_pre_election: true
I20260924 15:59:19.342955 18898 raft_consensus.cc:2454] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0292be8689d04d4ab112eafa591b6d70 in term 0.
I20260924 15:59:19.342957 18751 raft_consensus.cc:2454] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0292be8689d04d4ab112eafa591b6d70 in term 0.
I20260924 15:59:19.343346 18545 leader_election.cc:302] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0292be8689d04d4ab112eafa591b6d70, 4e5013313b77418e974d64a424969dea; no voters:
I20260924 15:59:19.343634 18971 raft_consensus.cc:2781] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:19.343750 18971 raft_consensus.cc:482] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:19.343837 18971 raft_consensus.cc:3037] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:19.345854 18971 raft_consensus.cc:504] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.346503 18971 leader_election.cc:288] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [CANDIDATE]: Term 1 election: Requested vote from peers 4e5013313b77418e974d64a424969dea (127.16.233.130:39011), 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131:32795)
I20260924 15:59:19.346900 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b446961fd5254e2980f081b3fb8f207e" candidate_uuid: "0292be8689d04d4ab112eafa591b6d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5013313b77418e974d64a424969dea"
I20260924 15:59:19.346966 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b446961fd5254e2980f081b3fb8f207e" candidate_uuid: "0292be8689d04d4ab112eafa591b6d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aae1d12570a43868fb1b8b8344807ba"
I20260924 15:59:19.347169 18751 raft_consensus.cc:3037] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:19.347204 18898 raft_consensus.cc:3037] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:19.349527 18751 raft_consensus.cc:2454] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0292be8689d04d4ab112eafa591b6d70 in term 1.
I20260924 15:59:19.349668 18898 raft_consensus.cc:2454] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0292be8689d04d4ab112eafa591b6d70 in term 1.
I20260924 15:59:19.350061 18545 leader_election.cc:302] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0292be8689d04d4ab112eafa591b6d70, 4e5013313b77418e974d64a424969dea; no voters:
I20260924 15:59:19.350446 18971 raft_consensus.cc:2781] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:19.350931 18971 raft_consensus.cc:686] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [term 1 LEADER]: Becoming Leader. State: Replica: 0292be8689d04d4ab112eafa591b6d70, State: Running, Role: LEADER
I20260924 15:59:19.351325 18971 consensus_queue.cc:237] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } }
I20260924 15:59:19.355022 19018 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: TxnStatusTablet state changed. Reason: New leader 0292be8689d04d4ab112eafa591b6d70. Latest consensus state: current_term: 1 leader_uuid: "0292be8689d04d4ab112eafa591b6d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.355410 19018 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:59:19.356968 18454 catalog_manager.cc:5495] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0292be8689d04d4ab112eafa591b6d70 (127.16.233.129). New cstate: current_term: 1 leader_uuid: "0292be8689d04d4ab112eafa591b6d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } health_report { overall_health: UNKNOWN } } }
I20260924 15:59:19.357403 19020 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:59:19.357643 19020 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:59:19.377305 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.370755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.16.233.129:34341)
0924 15:59:19.371021 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.371027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.371148 (+ 121us) server_negotiation.cc:407] Connection header received
0924 15:59:19.371447 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.371451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.371532 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.371647 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:19.372211 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.372952 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.375057 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.375499 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.375601 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.376217 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:19.376743 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:19.376848 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.377008 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.377081 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 15:59:19.377494 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.370743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:34341 (local address 127.0.0.1:57136)
0924 15:59:19.371047 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:19.371069 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:19.371239 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:19.371706 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:19.371711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:19.371724 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:59:19.372066 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.372082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.373145 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.373151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.374909 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.374926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.375664 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.375670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.375927 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.376021 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:59:19.376858 (+ 837us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:59:19.376885 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 15:59:19.376981 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 15:59:19.377151 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 15:59:19.382972 18751 raft_consensus.cc:1261] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea [term 1 FOLLOWER]: Refusing update from remote peer 0292be8689d04d4ab112eafa591b6d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:19.383049 18898 raft_consensus.cc:1261] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Refusing update from remote peer 0292be8689d04d4ab112eafa591b6d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:19.383713 19018 consensus_queue.cc:1035] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:19.384474 18971 consensus_queue.cc:1035] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:19.394825 19017 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: TxnStatusTablet state changed. Reason: New leader 0292be8689d04d4ab112eafa591b6d70. Latest consensus state: current_term: 1 leader_uuid: "0292be8689d04d4ab112eafa591b6d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.395164 19017 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:19.396983 19023 mvcc.cc:205] Tried to move back new op lower bound from 7332927731224551424 to 7332927731106533376. Current Snapshot: MvccSnapshot[applied={T|T < 7332927731224551424}]
I20260924 15:59:19.398445 19026 mvcc.cc:205] Tried to move back new op lower bound from 7332927731224551424 to 7332927731106533376. Current Snapshot: MvccSnapshot[applied={T|T < 7332927731224551424}]
I20260924 15:59:19.398149 18972 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: TxnStatusTablet state changed. Reason: New leader 0292be8689d04d4ab112eafa591b6d70. Latest consensus state: current_term: 1 leader_uuid: "0292be8689d04d4ab112eafa591b6d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.399046 18972 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:19.406973 19018 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0292be8689d04d4ab112eafa591b6d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.407444 19018 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:59:19.412070 19018 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0292be8689d04d4ab112eafa591b6d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.412446 19018 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 0292be8689d04d4ab112eafa591b6d70: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:59:19.417757 19017 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0292be8689d04d4ab112eafa591b6d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.418285 19017 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 0aae1d12570a43868fb1b8b8344807ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:19.419235 18972 tablet_replica.cc:439] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0292be8689d04d4ab112eafa591b6d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 } } peers { permanent_uuid: "4e5013313b77418e974d64a424969dea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39011 } } peers { permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 } } }
I20260924 15:59:19.419737 18972 tablet_replica.cc:442] T b446961fd5254e2980f081b3fb8f207e P 4e5013313b77418e974d64a424969dea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:19.429630 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.421676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.16.233.130:39011)
0924 15:59:19.421898 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.421905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.423079 (+ 1174us) server_negotiation.cc:407] Connection header received
0924 15:59:19.423305 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.423312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.423497 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.423683 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:19.424128 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.425279 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.426725 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.427376 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.427515 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.427898 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:19.428828 (+ 930us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:19.428971 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.429050 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.429242 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 15:59:19.450546 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.442708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:34341 (local address 127.16.233.130:58685)
0924 15:59:19.443826 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:19.443842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:19.444003 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:19.444423 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:19.444426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:19.444436 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:59:19.445045 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.445055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.446806 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.446810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.448159 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.448172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.449255 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.449261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.449425 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.449504 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:59:19.449955 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:59:19.449965 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 15:59:19.450026 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:59:19.450142 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":570,"threads_started":1}
I20260924 15:59:19.450556 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.442429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:58685 (local address 127.16.233.129:34341)
0924 15:59:19.442699 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.442706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.443969 (+ 1263us) server_negotiation.cc:407] Connection header received
0924 15:59:19.444176 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.444183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.444298 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.444403 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:19.445221 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.446606 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.448316 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.449027 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.449169 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.449690 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:19.449846 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:19.449957 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.450077 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.450176 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 15:59:19.461714 18898 raft_consensus.cc:1261] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Refusing update from remote peer 4e5013313b77418e974d64a424969dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:19.461871 18613 raft_consensus.cc:1261] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Refusing update from remote peer 4e5013313b77418e974d64a424969dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:19.462718 19002 consensus_queue.cc:1035] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aae1d12570a43868fb1b8b8344807ba" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:19.463122 18972 consensus_queue.cc:1035] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0292be8689d04d4ab112eafa591b6d70" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 34341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:19.469040 19024 mvcc.cc:205] Tried to move back new op lower bound from 7332927731548041216 to 7332927730303795200. Current Snapshot: MvccSnapshot[applied={T|T < 7332927731548041216}]
I20260924 15:59:19.511451 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.498114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:60475 (local address 127.16.233.130:39011)
0924 15:59:19.498338 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.498347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.503149 (+ 4802us) server_negotiation.cc:407] Connection header received
0924 15:59:19.503407 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.503414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.503515 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.503582 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:19.505085 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.506825 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.508952 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.509844 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.509975 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.510637 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:19.510805 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:19.510879 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.511104 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.511179 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 15:59:19.511792 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.496289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39011 (local address 127.16.233.129:60475)
0924 15:59:19.503003 (+ 6714us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:19.503025 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:19.503220 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:19.504128 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:19.504132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:19.504158 (+ 26us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:59:19.504772 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.504788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.506958 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.506962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.508776 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:19.508790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.510136 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.510143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:19.510375 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.510464 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:59:19.510901 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:59:19.510914 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:59:19.511007 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 15:59:19.511167 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6475,"thread_start_us":6241,"threads_started":1}
I20260924 15:59:19.560987 18452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:19.561271 18452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:19.565683 18452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccf3eba8a1be4cfb9dc72fd60158adfe: Sending DeleteTablet for 3 replicas of tablet 5ee093bf37a84fa4b464744a9e5fc1ec
I20260924 15:59:19.566823 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee093bf37a84fa4b464744a9e5fc1ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:19 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:37566
I20260924 15:59:19.567101 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee093bf37a84fa4b464744a9e5fc1ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:19 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:36808
I20260924 15:59:19.568138 18872 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee093bf37a84fa4b464744a9e5fc1ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:19 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:37006
I20260924 15:59:19.579771 19045 tablet_replica.cc:330] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: stopping tablet replica
I20260924 15:59:19.581308 19045 raft_consensus.cc:2229] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:19.581707 19045 raft_consensus.cc:2258] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:19.583295 19045 ts_tablet_manager.cc:1905] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:19.587219 19046 tablet_replica.cc:330] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: stopping tablet replica
I20260924 15:59:19.587441 19046 raft_consensus.cc:2229] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:19.587700 19046 raft_consensus.cc:2258] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:19.589123 19046 ts_tablet_manager.cc:1905] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:19.590605 19044 tablet_replica.cc:330] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: stopping tablet replica
I20260924 15:59:19.590876 19044 raft_consensus.cc:2229] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:59:19.591540 19044 raft_consensus.cc:2258] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:19.592698 19044 ts_tablet_manager.cc:1905] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:19.597267 19046 ts_tablet_manager.cc:1918] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:59:19.597409 19046 log.cc:1196] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/5ee093bf37a84fa4b464744a9e5fc1ec
I20260924 15:59:19.597915 19046 ts_tablet_manager.cc:1939] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0aae1d12570a43868fb1b8b8344807ba: Deleting consensus metadata
I20260924 15:59:19.599143 18441 catalog_manager.cc:4841] TS 0aae1d12570a43868fb1b8b8344807ba (127.16.233.131:32795): tablet 5ee093bf37a84fa4b464744a9e5fc1ec (table test-table [id=10e2b92516494c2581b5505529a6fbe7]) successfully deleted
I20260924 15:59:19.600011 19045 ts_tablet_manager.cc:1918] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:59:19.600471 19045 log.cc:1196] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/5ee093bf37a84fa4b464744a9e5fc1ec
I20260924 15:59:19.601377 19045 ts_tablet_manager.cc:1939] T 5ee093bf37a84fa4b464744a9e5fc1ec P 0292be8689d04d4ab112eafa591b6d70: Deleting consensus metadata
I20260924 15:59:19.601439 19044 ts_tablet_manager.cc:1918] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:59:19.601583 19044 log.cc:1196] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/5ee093bf37a84fa4b464744a9e5fc1ec
I20260924 15:59:19.602061 19044 ts_tablet_manager.cc:1939] T 5ee093bf37a84fa4b464744a9e5fc1ec P 4e5013313b77418e974d64a424969dea: Deleting consensus metadata
I20260924 15:59:19.602751 18441 catalog_manager.cc:4841] TS 0292be8689d04d4ab112eafa591b6d70 (127.16.233.129:34341): tablet 5ee093bf37a84fa4b464744a9e5fc1ec (table test-table [id=10e2b92516494c2581b5505529a6fbe7]) successfully deleted
I20260924 15:59:19.603139 18440 catalog_manager.cc:4841] TS 4e5013313b77418e974d64a424969dea (127.16.233.130:39011): tablet 5ee093bf37a84fa4b464744a9e5fc1ec (table test-table [id=10e2b92516494c2581b5505529a6fbe7]) successfully deleted
Sep 24 15:59:19 dist-test-slave-xn4h krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:19.623401 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.580354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.16.233.129:34341)
0924 15:59:19.580536 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.580542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.592280 (+ 11738us) server_negotiation.cc:407] Connection header received
0924 15:59:19.594441 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.594447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.594552 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.594630 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:19.603481 (+ 8851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.604674 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.606791 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.607510 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.607617 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.613839 (+ 6222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:19.613860 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:19.613881 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:19.613940 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:19.618743 (+ 4803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.619987 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.619998 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.620002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.620287 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.620858 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.620864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.620868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.621585 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:19.621948 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.622730 (+ 782us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.623013 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 15:59:19.626027 18658 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57148
I20260924 15:59:19.656196 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.638073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.16.233.190:41823)
0924 15:59:19.638373 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.638380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.638915 (+ 535us) server_negotiation.cc:407] Connection header received
0924 15:59:19.639191 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.639197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.639290 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.639411 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:19.641163 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.642267 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.644147 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.644825 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.644967 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.648315 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:19.648325 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:19.648330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:19.648385 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:19.652542 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.653660 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.653667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.653671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.653848 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.654372 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.654377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.654381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.654781 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:19.655066 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.655532 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.655788 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 15:59:19.656920 18452 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50064
I20260924 15:59:19.659756 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 18526
I20260924 15:59:19.684696 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.661076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.16.233.129:34341)
0924 15:59:19.661290 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.661295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.666948 (+ 5653us) server_negotiation.cc:407] Connection header received
0924 15:59:19.667154 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.667159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.667262 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.667401 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:19.669180 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.670490 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.672361 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.673097 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.673228 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.676511 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:19.676518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:19.676524 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:19.676569 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:19.681114 (+ 4545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.682008 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.682015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.682019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.682395 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.682832 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.682837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.682841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.683371 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:19.683727 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.683999 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.684267 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 15:59:19.685365 18658 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57164
W20260924 15:59:19.686005 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 18526: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 15:59:19.686205 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 18526
I20260924 15:59:19.709036 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 18667
Sep 24 15:59:19 dist-test-slave-xn4h krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 15:59:19.727636 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.709867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.16.233.130:39011)
0924 15:59:19.710062 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.710068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.710092 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:59:19.710249 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.710256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.710375 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.710492 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:19.712242 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.713457 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.715242 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.715959 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.716098 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.720636 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:19.720663 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:19.720686 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:19.720742 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:19.724708 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.725399 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.725409 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.725413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.725611 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.725954 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.725960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.725963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.726536 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:19.726845 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.727021 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.727271 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 15:59:19.728473 18797 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43384
W20260924 15:59:19.729154 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 18667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 15:59:19.729351 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 18667
I20260924 15:59:19.748672 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 18807
Sep 24 15:59:19 dist-test-slave-xn4h krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 15:59:19.763645 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:19.749457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.16.233.131:32795)
0924 15:59:19.749946 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:19.749950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:19.749967 (+ 17us) server_negotiation.cc:407] Connection header received
0924 15:59:19.750022 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:19.750025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:19.750085 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:19.750177 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:19.751458 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.752175 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.753406 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:19.753857 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:19.753953 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:19.758064 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:19.758080 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:19.758096 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:19.758135 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:19.761144 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.761762 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.761769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.761772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.761909 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:19.762210 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:19.762213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:19.762215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:19.762645 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:19.762872 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:19.763086 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 15:59:19.763257 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":275,"threads_started":1}
W20260924 15:59:19.764303 18939 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56126
W20260924 15:59:19.764926 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 18807: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 15:59:19.765091 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 18807
I20260924 15:59:19.787359 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 18422
I20260924 15:59:19.788908 18495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:19.905941 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 18422
2026-09-24T15:59:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5149 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:19 dist-test-slave-xn4h krb5kdc[19056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:20 dist-test-slave-xn4h krb5kdc[19056](info): set up 2 sockets
Sep 24 15:59:20 dist-test-slave-xn4h krb5kdc[19056](info): commencing operation
krb5kdc: starting...
W20260924 15:59:20.991940 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:21 dist-test-slave-xn4h krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:21.194571 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36343
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:21.515558 19069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:21.515836 19069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:21.516011 19069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:21.523484 19069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:21.523609 19069 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:59:21.523717 19069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:21.523808 19069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:21.523900 19069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:21.523998 19069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:21.533952 19069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36343
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:21.535287 19069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:21.536640 19069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:21.548404 19074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:21.548344 19077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:21.548367 19075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:21.549183 19069 server_base.cc:1034] running on GCE node
I20260924 15:59:21.549980 19069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:21.551416 19069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:21.552652 19069 hybrid_clock.cc:648] HybridClock initialized: now 1790265561552585 us; error 55 us; skew 500 ppm
I20260924 15:59:21.552963 19069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:21 dist-test-slave-xn4h krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265561, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:21.556049 19069 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:21.557394 19069 webserver.cc:460] Webserver started at http://127.16.233.190:44411/ using document root <none> and password file <none>
I20260924 15:59:21.558035 19069 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:21.558120 19069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:21.558459 19069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:21.561306 19069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "8b487575036240b6a285d43f3f42440d"
format_stamp: "Formatted at 2026-09-24 15:59:21 on dist-test-slave-xn4h"
server_key: "317c9062b27f737c9d5a39c4138cf628"
server_key_iv: "b9e9894bf94188d375f65a54cea3b13a"
server_key_version: "encryptionkey@0"
I20260924 15:59:21.562006 19069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "8b487575036240b6a285d43f3f42440d"
format_stamp: "Formatted at 2026-09-24 15:59:21 on dist-test-slave-xn4h"
server_key: "317c9062b27f737c9d5a39c4138cf628"
server_key_iv: "b9e9894bf94188d375f65a54cea3b13a"
server_key_version: "encryptionkey@0"
I20260924 15:59:21.567132 19069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 15:59:21.570755 19084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:21.572129 19069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 15:59:21.572347 19069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "8b487575036240b6a285d43f3f42440d"
format_stamp: "Formatted at 2026-09-24 15:59:21 on dist-test-slave-xn4h"
server_key: "317c9062b27f737c9d5a39c4138cf628"
server_key_iv: "b9e9894bf94188d375f65a54cea3b13a"
server_key_version: "encryptionkey@0"
I20260924 15:59:21.572580 19069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:21.593132 19069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:21.597752 19069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:21.598057 19069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:21.620128 19069 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:36343
I20260924 15:59:21.620136 19146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:36343 every 8 connection(s)
I20260924 15:59:21.621556 19069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:21.625414 19147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:21.631705 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 19069
I20260924 15:59:21.631968 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:21.632602 17318 external_mini_cluster.cc:1428] Setting key 1b56ba4898555956b77013ee39a6dc02
I20260924 15:59:21.637063 19147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d: Bootstrap starting.
I20260924 15:59:21.640278 19147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:21.641325 19147 log.cc:824] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:21.644259 19147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d: No bootstrap required, opened a new log
Sep 24 15:59:21 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:21.652539 19147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b487575036240b6a285d43f3f42440d" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } }
I20260924 15:59:21.652962 19147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:21.653079 19147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b487575036240b6a285d43f3f42440d, State: Initialized, Role: FOLLOWER
I20260924 15:59:21.653846 19147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b487575036240b6a285d43f3f42440d" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } }
I20260924 15:59:21.654132 19147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:21.654273 19147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:21.654511 19147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:21.655356 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:21.634767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.16.233.190:36343)
0924 15:59:21.635579 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:21.635591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:21.635640 (+ 49us) server_negotiation.cc:407] Connection header received
0924 15:59:21.637358 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:21.637381 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:21.637775 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:21.638169 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:21.639656 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:21.641046 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:21.642367 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:21.643032 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:21.643224 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:21.647661 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:21.647672 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:21.647687 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:21.647739 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:21.650890 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:21.651690 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:21.651702 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:21.651709 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:21.651877 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:21.652281 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:21.652285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:21.652287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:21.652819 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:21.653161 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:21.653645 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 15:59:21.654078 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":362,"threads_started":1}
I20260924 15:59:21.656944 19147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b487575036240b6a285d43f3f42440d" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } }
I20260924 15:59:21.657467 19147 leader_election.cc:302] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b487575036240b6a285d43f3f42440d; no voters:
I20260924 15:59:21.658176 19147 leader_election.cc:288] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:21.658324 19152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:21.658989 19152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [term 1 LEADER]: Becoming Leader. State: Replica: 8b487575036240b6a285d43f3f42440d, State: Running, Role: LEADER
I20260924 15:59:21.659543 19152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b487575036240b6a285d43f3f42440d" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } }
I20260924 15:59:21.660446 19147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:21.662321 19153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b487575036240b6a285d43f3f42440d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b487575036240b6a285d43f3f42440d" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } } }
I20260924 15:59:21.662688 19153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:21.663273 19154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b487575036240b6a285d43f3f42440d. Latest consensus state: current_term: 1 leader_uuid: "8b487575036240b6a285d43f3f42440d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b487575036240b6a285d43f3f42440d" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } } }
I20260924 15:59:21.663614 19154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:21.665290 19157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:21.673898 19157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:21.682767 19157 catalog_manager.cc:1347] Generated new cluster ID: 36a9fc2ceb63459e9959ae17be917667
I20260924 15:59:21.682914 19157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:21.690879 19157 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:21.692152 19157 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:21.701332 19157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d: Generated new TSK 0
I20260924 15:59:21.702252 19157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 15:59:21.756186 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36343
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:22.085393 19173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:22.085829 19173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:22.086030 19173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:22.093544 19173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:22.093717 19173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:22.093869 19173 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:59:22.093963 19173 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:59:22.094099 19173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:22.104233 19173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.190:36343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:22.105695 19173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:22.107046 19173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:22.121526 19178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:22.121680 19179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:22.122118 19181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:22.122393 19173 server_base.cc:1034] running on GCE node
I20260924 15:59:22.123332 19173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:22.124369 19173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:22.125597 19173 hybrid_clock.cc:648] HybridClock initialized: now 1790265562125529 us; error 55 us; skew 500 ppm
I20260924 15:59:22.125967 19173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:22 dist-test-slave-xn4h krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265562, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:22.129448 19173 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 15:59:22.131100 19173 webserver.cc:460] Webserver started at http://127.16.233.129:38041/ using document root <none> and password file <none>
I20260924 15:59:22.131806 19173 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:22.131950 19173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:22.132300 19173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:22.135406 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "841f8fa6250241d38d72dd5d8126f08c"
format_stamp: "Formatted at 2026-09-24 15:59:22 on dist-test-slave-xn4h"
server_key: "213ba3ca270f85206410b5616a1b2477"
server_key_iv: "e0272e460592522c2c3873df7742212a"
server_key_version: "encryptionkey@0"
I20260924 15:59:22.136161 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "841f8fa6250241d38d72dd5d8126f08c"
format_stamp: "Formatted at 2026-09-24 15:59:22 on dist-test-slave-xn4h"
server_key: "213ba3ca270f85206410b5616a1b2477"
server_key_iv: "e0272e460592522c2c3873df7742212a"
server_key_version: "encryptionkey@0"
I20260924 15:59:22.141273 19173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 15:59:22.144766 19188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:22.146090 19173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 15:59:22.146279 19173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "841f8fa6250241d38d72dd5d8126f08c"
format_stamp: "Formatted at 2026-09-24 15:59:22 on dist-test-slave-xn4h"
server_key: "213ba3ca270f85206410b5616a1b2477"
server_key_iv: "e0272e460592522c2c3873df7742212a"
server_key_version: "encryptionkey@0"
I20260924 15:59:22.146586 19173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:22.168807 19173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:22.173002 19173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:22.173322 19173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:22.176462 19173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:22.176594 19173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:22.176720 19173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:22.176828 19173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:59:22 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265562, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:22.209843 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:22.176678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:57649 (local address 127.16.233.190:36343)
0924 15:59:22.191779 (+ 15101us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:22.191788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:22.191825 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:59:22.191929 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:22.191937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:22.192042 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:22.192213 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:22.193969 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.194914 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.197493 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.197884 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.198016 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:22.202278 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:22.202283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:22.202287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:22.202368 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:22.205883 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:22.206891 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:22.206900 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:22.206906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:22.207101 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:22.207564 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:22.207572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:22.207578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:22.208039 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:22.208231 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:22.208930 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 15:59:22.209285 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14945,"thread_start_us":15664,"threads_started":1}
I20260924 15:59:22.210497 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:22.177483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.16.233.129:57649)
0924 15:59:22.178615 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:22.178672 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:22.181445 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:22.192541 (+ 11096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:22.192551 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:22.193157 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:22.193767 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:22.193790 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.195145 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.195149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:22.197341 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:22.197354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.198123 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.198131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:22.198446 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:22.199326 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:22.199364 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:22.202054 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:22.206030 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:22.206039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:22.206058 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:22.206713 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:22.207233 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:22.207242 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:22.207245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:22.207437 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:22.208173 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:22.208178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:22.208540 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 15:59:22.208762 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 15:59:22.209176 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":510,"threads_started":1}
I20260924 15:59:22.231917 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:22.217333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:58279 (local address 127.16.233.190:36343)
0924 15:59:22.217514 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:22.217520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:22.218365 (+ 845us) server_negotiation.cc:407] Connection header received
0924 15:59:22.218539 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:22.218547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:22.218675 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:22.218819 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:22.219866 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.221344 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.222863 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.223494 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.223633 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:22.225874 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:22.225879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:22.225883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:22.225913 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:22.229155 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:22.229813 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:22.229819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:22.229823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:22.229965 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:22.230362 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:22.230366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:22.230368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:22.230620 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:22.230825 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:22.231426 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 15:59:22.231675 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 15:59:22.232434 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:22.217504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.16.233.129:58279)
0924 15:59:22.218198 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:22.218236 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:22.218413 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:22.218819 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:22.218822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:22.219220 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:22.219660 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:22.219673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.221495 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.221498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:22.222656 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:22.222668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.223657 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.223665 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:22.223805 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:22.224445 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:22.224461 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:22.225627 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:22.229289 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:22.229299 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:22.229303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:22.229685 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:22.230072 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:22.230078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:22.230081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:22.230208 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:22.230801 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:22.230806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:22.231105 (+ 299us) client_negotiation.cc:769] Sending connection context
0924 15:59:22.231301 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 15:59:22.231768 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":422,"threads_started":1}
I20260924 15:59:22.237084 19173 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:41167
I20260924 15:59:22.237130 19309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:41167 every 8 connection(s)
I20260924 15:59:22.238461 19173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:22.240510 19310 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36343
I20260924 15:59:22.240700 19310 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:22.241299 19310 heartbeater.cc:511] Master 127.16.233.190:36343 requested a full tablet report, sending...
I20260924 15:59:22.243157 19102 ts_manager.cc:194] Registered new tserver with Master: 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167)
I20260924 15:59:22.244601 19102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:57649
I20260924 15:59:22.247133 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 19173
I20260924 15:59:22.247344 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:22.247946 17318 external_mini_cluster.cc:1428] Setting key 0b1189e00d25af0a4e3a9f4b40310e5d
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 15:59:22.289544 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36343
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:22.607482 19314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:22.607750 19314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:22.607852 19314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:22.615109 19314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:22.615239 19314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:22.615386 19314 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:59:22.615447 19314 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:59:22.615516 19314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:22.628684 19314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.190:36343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:22.630244 19314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:22.631839 19314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:22.646806 19322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:22.646776 19320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:22.646879 19319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:22.651412 19314 server_base.cc:1034] running on GCE node
I20260924 15:59:22.652046 19314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:22.652994 19314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:22.654227 19314 hybrid_clock.cc:648] HybridClock initialized: now 1790265562654177 us; error 64 us; skew 500 ppm
I20260924 15:59:22.654564 19314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:22 dist-test-slave-xn4h krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265562, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:22.657698 19314 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 15:59:22.659147 19314 webserver.cc:460] Webserver started at http://127.16.233.130:33555/ using document root <none> and password file <none>
I20260924 15:59:22.659825 19314 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:22.659924 19314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:22.660199 19314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:22.663127 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d"
format_stamp: "Formatted at 2026-09-24 15:59:22 on dist-test-slave-xn4h"
server_key: "fd91a36b488e3c6b7b240617be0bc05c"
server_key_iv: "2aeb9b1c17c7c3ac02a345a5fb2191a8"
server_key_version: "encryptionkey@0"
I20260924 15:59:22.663892 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d"
format_stamp: "Formatted at 2026-09-24 15:59:22 on dist-test-slave-xn4h"
server_key: "fd91a36b488e3c6b7b240617be0bc05c"
server_key_iv: "2aeb9b1c17c7c3ac02a345a5fb2191a8"
server_key_version: "encryptionkey@0"
I20260924 15:59:22.669129 19314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 15:59:22.672726 19329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:22.674015 19314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 15:59:22.674193 19314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d"
format_stamp: "Formatted at 2026-09-24 15:59:22 on dist-test-slave-xn4h"
server_key: "fd91a36b488e3c6b7b240617be0bc05c"
server_key_iv: "2aeb9b1c17c7c3ac02a345a5fb2191a8"
server_key_version: "encryptionkey@0"
I20260924 15:59:22.674450 19314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:22.706378 19314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:22.710846 19314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:22.711146 19314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:22.714003 19314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:22.714170 19314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:22.714341 19314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:22.714437 19314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 24 15:59:22 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265562, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:22.737356 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:22.714521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:45299 (local address 127.16.233.190:36343)
0924 15:59:22.714768 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:22.714779 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:22.716116 (+ 1337us) server_negotiation.cc:407] Connection header received
0924 15:59:22.718383 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:22.718392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:22.718510 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:22.718629 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:22.720964 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.722053 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.724496 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.724939 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.725077 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:22.729797 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:22.729807 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:22.729814 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:22.729863 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:22.733237 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:22.734345 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:22.734353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:22.734359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:22.734525 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:22.734982 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:22.734990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:22.734996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:22.735485 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:22.735673 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:22.736402 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 15:59:22.736711 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 15:59:22.738126 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:22.715127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.16.233.130:45299)
0924 15:59:22.715949 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:22.715999 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:22.718034 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:22.718906 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:22.718917 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:22.719673 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:22.720767 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:22.720799 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.722220 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.722226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:22.724346 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:22.724359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.725183 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.725201 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:22.725437 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:22.726202 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:22.726225 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:22.729451 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:22.733399 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:22.733411 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:22.733432 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:22.734161 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:22.734649 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:22.734661 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:22.734665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:22.734856 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:22.735636 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:22.735642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:22.735989 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 15:59:22.736222 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 15:59:22.736607 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":387,"threads_started":1}
I20260924 15:59:22.762075 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:22.744109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:60889 (local address 127.16.233.190:36343)
0924 15:59:22.744299 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:22.744303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:22.745103 (+ 800us) server_negotiation.cc:407] Connection header received
0924 15:59:22.745462 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:22.745469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:22.745553 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:22.745657 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:22.746748 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.748042 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.750418 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.751240 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.751436 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:22.754752 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:22.754770 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:22.754778 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:22.754838 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:22.758665 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:22.759784 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:22.759792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:22.759798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:22.759992 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:22.760528 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:22.760534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:22.760538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:22.760965 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:22.761121 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:22.761524 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 15:59:22.761731 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 15:59:22.762210 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:22.744244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.16.233.130:60889)
0924 15:59:22.744999 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:22.745019 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:22.745223 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:22.745681 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:22.745685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:22.746169 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:22.746571 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:22.746589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.748207 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.748215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:22.750258 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:22.750273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:22.751735 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:22.751744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:22.752041 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:22.752908 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:22.752936 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:22.754453 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:22.758865 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:22.758873 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:22.758877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:22.759611 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:22.760139 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:22.760145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:22.760149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:22.760361 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:22.761091 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:22.761096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:22.761273 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 15:59:22.761408 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 15:59:22.761731 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":371,"threads_started":1}
I20260924 15:59:22.767395 19314 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:39853
I20260924 15:59:22.767441 19449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:39853 every 8 connection(s)
I20260924 15:59:22.768621 19314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:22.770740 19450 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36343
I20260924 15:59:22.770978 19450 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:22.771534 19450 heartbeater.cc:511] Master 127.16.233.190:36343 requested a full tablet report, sending...
I20260924 15:59:22.772888 19102 ts_manager.cc:194] Registered new tserver with Master: ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853)
I20260924 15:59:22.773680 19102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:45299
I20260924 15:59:22.778118 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 19314
I20260924 15:59:22.778296 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:22.778827 17318 external_mini_cluster.cc:1428] Setting key d7bb894162a41641510e2c3d9421ea76
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 15:59:22.821231 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36343
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:23.140880 19454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:23.141198 19454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:23.141371 19454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:23.148659 19454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:23.148814 19454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:23.148962 19454 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:59:23.149049 19454 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:59:23.149180 19454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:23.159266 19454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.190:36343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:23.160652 19454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:23.162071 19454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:23.175213 19462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:23.175134 19460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:23.175264 19459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:23.176353 19454 server_base.cc:1034] running on GCE node
I20260924 15:59:23.176986 19454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:23.178084 19454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:23.179317 19454 hybrid_clock.cc:648] HybridClock initialized: now 1790265563179260 us; error 48 us; skew 500 ppm
I20260924 15:59:23.179682 19454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:23 dist-test-slave-xn4h krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265563, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:23.182952 19454 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 15:59:23.184535 19454 webserver.cc:460] Webserver started at http://127.16.233.131:32825/ using document root <none> and password file <none>
I20260924 15:59:23.185254 19454 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:23.185391 19454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:23.185745 19454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:23.188750 19454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "87916da5b0e74a26add7c8341299fdc5"
format_stamp: "Formatted at 2026-09-24 15:59:23 on dist-test-slave-xn4h"
server_key: "7c339d8f1579f7be44294f29756286e4"
server_key_iv: "c633bbae5a1f36fb94eb1a8c111c02c2"
server_key_version: "encryptionkey@0"
I20260924 15:59:23.189522 19454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "87916da5b0e74a26add7c8341299fdc5"
format_stamp: "Formatted at 2026-09-24 15:59:23 on dist-test-slave-xn4h"
server_key: "7c339d8f1579f7be44294f29756286e4"
server_key_iv: "c633bbae5a1f36fb94eb1a8c111c02c2"
server_key_version: "encryptionkey@0"
I20260924 15:59:23.194782 19454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 15:59:23.198218 19469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:23.199615 19454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:23.199800 19454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "87916da5b0e74a26add7c8341299fdc5"
format_stamp: "Formatted at 2026-09-24 15:59:23 on dist-test-slave-xn4h"
server_key: "7c339d8f1579f7be44294f29756286e4"
server_key_iv: "c633bbae5a1f36fb94eb1a8c111c02c2"
server_key_version: "encryptionkey@0"
I20260924 15:59:23.199998 19454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:23.214331 19454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:23.218695 19454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:23.218995 19454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:23.221720 19454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:23.221876 19454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:23.222059 19454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:23.222203 19454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:59:23 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265563, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:23.246682 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.223210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:42459 (local address 127.16.233.190:36343)
0924 15:59:23.223457 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:23.223464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:23.223774 (+ 310us) server_negotiation.cc:407] Connection header received
0924 15:59:23.225824 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:23.225830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:23.225918 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:23.226038 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:23.228155 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.229477 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.231591 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.232355 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.232505 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.238375 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:23.238383 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:23.238389 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:23.238437 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:23.242171 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:23.243488 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:23.243497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:23.243504 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:23.243688 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:23.244293 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:23.244301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:23.244307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:23.244767 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:23.244953 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:23.245808 (+ 855us) server_negotiation.cc:299] Negotiation successful
0924 15:59:23.246147 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 15:59:23.247552 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.222703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.16.233.131:42459)
0924 15:59:23.223597 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:23.223651 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:23.225538 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:23.226286 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:23.226294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:23.227061 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:23.227928 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.227966 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.229666 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.229674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.231403 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.231424 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.232648 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.232662 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.232927 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.233718 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:23.233760 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:23.238011 (+ 4251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:23.242424 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:23.242439 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:23.242462 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:23.243298 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:23.243818 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:23.243823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:23.243827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:23.244130 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:23.244922 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:23.244932 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:23.245292 (+ 360us) client_negotiation.cc:769] Sending connection context
0924 15:59:23.245523 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 15:59:23.245942 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":495,"threads_started":1}
I20260924 15:59:23.248229 19310 heartbeater.cc:503] Master 127.16.233.190:36343 was elected leader, sending a full tablet report...
I20260924 15:59:23.269063 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.253716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:52945 (local address 127.16.233.190:36343)
0924 15:59:23.253941 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:23.253950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:23.254555 (+ 605us) server_negotiation.cc:407] Connection header received
0924 15:59:23.254844 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:23.254851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:23.254937 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:23.255060 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:23.256969 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.257893 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.260015 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.260546 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.260675 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.262738 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:23.262744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:23.262750 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:23.262790 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:23.265878 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:23.267028 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:23.267036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:23.267047 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:23.267215 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:23.267682 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:23.267685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:23.267688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:23.267962 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:23.268080 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:23.268636 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 15:59:23.268814 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 15:59:23.269698 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.253722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.16.233.131:52945)
0924 15:59:23.254451 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:23.254470 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:23.254637 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:23.255261 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:23.255266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:23.256119 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:23.256773 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.256792 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.258039 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.258045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.259843 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.259867 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.260747 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.260754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.260947 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.261716 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:23.261742 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:23.262545 (+ 803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:23.266061 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:23.266067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:23.266072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:23.266864 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:23.267343 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:23.267350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:23.267356 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:23.267563 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:23.268067 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:23.268076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:23.268346 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 15:59:23.268537 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 15:59:23.269032 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":369,"threads_started":1}
I20260924 15:59:23.284518 19454 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:33639
I20260924 15:59:23.284565 19588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:33639 every 8 connection(s)
I20260924 15:59:23.285794 19454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:23.287974 19589 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36343
I20260924 15:59:23.288201 19589 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:23.288616 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 19454
I20260924 15:59:23.288793 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:23.288939 19589 heartbeater.cc:511] Master 127.16.233.190:36343 requested a full tablet report, sending...
I20260924 15:59:23.289294 17318 external_mini_cluster.cc:1428] Setting key 5619b7a53f53dd946e0365035f48acce
I20260924 15:59:23.290510 19102 ts_manager.cc:194] Registered new tserver with Master: 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639)
I20260924 15:59:23.291637 19102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:42459
I20260924 15:59:23.292529 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:59:23 dist-test-slave-xn4h krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:59:23 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:23.339403 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.325986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.16.233.190:36343)
0924 15:59:23.326228 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:23.326235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:23.326446 (+ 211us) server_negotiation.cc:407] Connection header received
0924 15:59:23.326664 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:23.326673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:23.326749 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:23.326860 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:23.328200 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.328929 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.330167 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.330655 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.330812 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.334471 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:23.334476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:23.334479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:23.334512 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:23.336834 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:23.337460 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:23.337466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:23.337470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:23.337603 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:23.337974 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:23.337979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:23.337982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:23.338255 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:23.338584 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:23.338826 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 15:59:23.338992 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 15:59:23.343673 19101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:59:23.350966 19101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:23.384943 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.366761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.16.233.130:39853)
0924 15:59:23.372796 (+ 6035us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:23.372810 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:23.377886 (+ 5076us) server_negotiation.cc:407] Connection header received
0924 15:59:23.378012 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:23.378019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:23.378221 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:23.378432 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:23.379129 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.380467 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.382910 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.383764 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.383958 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.384122 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:23.384397 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 15:59:23.384572 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5863,"thread_start_us":5718,"threads_started":1}
I20260924 15:59:23.385049 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.365865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39853 (local address 127.0.0.1:49482)
0924 15:59:23.377710 (+ 11845us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:23.377729 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:23.377855 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:23.378437 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:23.378447 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:23.378479 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:23.378973 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.378985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.380650 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.380655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.382765 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.382776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.383958 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.383965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.384146 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.384195 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:59:23.384363 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 15:59:23.384578 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11698,"mutex_wait_us":92,"thread_start_us":11516,"threads_started":1}
I20260924 15:59:23.385386 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.363458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.16.233.129:41167)
0924 15:59:23.374807 (+ 11349us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:23.374818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:23.375952 (+ 1134us) server_negotiation.cc:407] Connection header received
0924 15:59:23.376096 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:23.376108 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:23.376427 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:23.376660 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:23.377763 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.379431 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.382198 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.383756 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.383927 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.384118 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:23.384712 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 15:59:23.384893 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11190,"thread_start_us":11052,"threads_started":1}
I20260924 15:59:23.385967 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.364198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.16.233.131:33639)
0924 15:59:23.365145 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:23.365159 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:23.376908 (+ 11749us) server_negotiation.cc:407] Connection header received
0924 15:59:23.377233 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:23.377255 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:23.377497 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:23.377664 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:23.378957 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.380311 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.382816 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.384201 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.384361 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.384512 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:23.385462 (+ 950us) server_negotiation.cc:299] Negotiation successful
0924 15:59:23.385632 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":456,"threads_started":1}
I20260924 15:59:23.386071 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.363269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:41167 (local address 127.0.0.1:52608)
0924 15:59:23.375383 (+ 12114us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:23.375439 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:23.375672 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:23.376720 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:23.376732 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:23.376783 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:23.377525 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.377545 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.379577 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.379583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.382020 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.382035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.384289 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.384296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.384494 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.384526 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 15:59:23.385597 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0924 15:59:23.385722 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11770,"mutex_wait_us":6,"thread_start_us":11609,"threads_started":1}
I20260924 15:59:23.386741 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.363394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33639 (local address 127.0.0.1:60494)
0924 15:59:23.376786 (+ 13392us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:23.376805 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:23.377019 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:23.378070 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:23.378073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:23.378134 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:23.378712 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.378724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.380486 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.380491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.382597 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.382611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.384725 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.384732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.384910 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.384939 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 15:59:23.385020 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:59:23.386280 (+ 1260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13245,"mutex_wait_us":77,"thread_start_us":13472,"threads_started":1}
I20260924 15:59:23.389183 19379 tablet_service.cc:1467] Processing CreateTablet for tablet a1aa55a533014f7289ad50c6533663c7 (DEFAULT_TABLE table=test-table [id=8c25a46b5e76443fb335d98efa3e7c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:23.389508 19519 tablet_service.cc:1467] Processing CreateTablet for tablet a1aa55a533014f7289ad50c6533663c7 (DEFAULT_TABLE table=test-table [id=8c25a46b5e76443fb335d98efa3e7c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:23.389825 19239 tablet_service.cc:1467] Processing CreateTablet for tablet a1aa55a533014f7289ad50c6533663c7 (DEFAULT_TABLE table=test-table [id=8c25a46b5e76443fb335d98efa3e7c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:23.390695 19379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1aa55a533014f7289ad50c6533663c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:23.390785 19519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1aa55a533014f7289ad50c6533663c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:23.391050 19239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1aa55a533014f7289ad50c6533663c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:23.401726 19608 tablet_bootstrap.cc:492] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: Bootstrap starting.
I20260924 15:59:23.403620 19609 tablet_bootstrap.cc:492] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: Bootstrap starting.
I20260924 15:59:23.406256 19610 tablet_bootstrap.cc:492] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: Bootstrap starting.
I20260924 15:59:23.406677 19608 tablet_bootstrap.cc:654] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:23.407007 19609 tablet_bootstrap.cc:654] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:23.407980 19608 log.cc:824] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:23.408073 19609 log.cc:824] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:23.409294 19610 tablet_bootstrap.cc:654] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:23.410254 19610 log.cc:824] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:23.411139 19608 tablet_bootstrap.cc:492] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: No bootstrap required, opened a new log
I20260924 15:59:23.411139 19609 tablet_bootstrap.cc:492] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: No bootstrap required, opened a new log
I20260924 15:59:23.411401 19609 ts_tablet_manager.cc:1397] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 15:59:23.411397 19608 ts_tablet_manager.cc:1397] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 15:59:23.412859 19610 tablet_bootstrap.cc:492] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: No bootstrap required, opened a new log
I20260924 15:59:23.413110 19610 ts_tablet_manager.cc:1397] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 15:59:23.420065 19610 raft_consensus.cc:348] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } }
I20260924 15:59:23.420444 19610 raft_consensus.cc:374] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:23.420568 19610 raft_consensus.cc:729] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841f8fa6250241d38d72dd5d8126f08c, State: Initialized, Role: FOLLOWER
I20260924 15:59:23.421185 19610 consensus_queue.cc:260] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } }
I20260924 15:59:23.421977 19608 raft_consensus.cc:348] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } }
I20260924 15:59:23.422472 19608 raft_consensus.cc:374] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:23.422564 19608 raft_consensus.cc:729] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0e0695e2be4bd3b83cb79e1049fc4d, State: Initialized, Role: FOLLOWER
I20260924 15:59:23.423086 19610 ts_tablet_manager.cc:1428] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260924 15:59:23.423087 19609 raft_consensus.cc:348] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } }
I20260924 15:59:23.423267 19608 consensus_queue.cc:260] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } }
I20260924 15:59:23.423462 19609 raft_consensus.cc:374] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:23.423550 19609 raft_consensus.cc:729] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87916da5b0e74a26add7c8341299fdc5, State: Initialized, Role: FOLLOWER
I20260924 15:59:23.424222 19609 consensus_queue.cc:260] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } }
I20260924 15:59:23.424906 19608 ts_tablet_manager.cc:1428] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260924 15:59:23.425328 19450 heartbeater.cc:503] Master 127.16.233.190:36343 was elected leader, sending a full tablet report...
I20260924 15:59:23.425346 19589 heartbeater.cc:503] Master 127.16.233.190:36343 was elected leader, sending a full tablet report...
I20260924 15:59:23.425833 19609 ts_tablet_manager.cc:1428] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260924 15:59:23.490935 19311 tablet.cc:2369] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:59:23.520596 19451 tablet.cc:2369] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:59:23.537544 19590 tablet.cc:2369] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:59:23.592141 19614 raft_consensus.cc:482] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:23.592471 19614 raft_consensus.cc:504] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } }
I20260924 15:59:23.594158 19614 leader_election.cc:288] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639), ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853)
I20260924 15:59:23.601696 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.594757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:51675 (local address 127.16.233.131:33639)
0924 15:59:23.594983 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:23.594990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:23.595589 (+ 599us) server_negotiation.cc:407] Connection header received
0924 15:59:23.595833 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:23.595837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:23.595899 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:23.596001 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:23.596851 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.597838 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.599729 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.600716 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.600888 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.600963 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:23.601285 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 15:59:23.601427 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 15:59:23.601845 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.594568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33639 (local address 127.16.233.129:51675)
0924 15:59:23.595438 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:23.595457 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:23.595613 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:23.596179 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:23.596184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:23.596220 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:23.596690 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.596705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.597992 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.597996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.599580 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.599593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.600947 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.600953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.601105 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.601139 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 15:59:23.601245 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:59:23.601391 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":32,"thread_start_us":461,"threads_started":1}
I20260924 15:59:23.602672 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.594806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:49051 (local address 127.16.233.130:39853)
0924 15:59:23.595019 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:23.595028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:23.595644 (+ 616us) server_negotiation.cc:407] Connection header received
0924 15:59:23.595754 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:23.595763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:23.595861 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:23.595965 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:23.596564 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.597624 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.600362 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.601504 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.601670 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.601751 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:23.602253 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 15:59:23.602423 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:59:23.602852 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.594769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39853 (local address 127.16.233.129:49051)
0924 15:59:23.595437 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:23.595462 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:23.595646 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:23.596003 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:23.596008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:23.596034 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:23.596391 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.596402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.597769 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.597778 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.600189 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.600204 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.601859 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.601869 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.602048 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.602083 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 15:59:23.602178 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 15:59:23.602269 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":51,"spinlock_wait_cycles":33664,"thread_start_us":389,"threads_started":1}
I20260924 15:59:23.603601 19544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aa55a533014f7289ad50c6533663c7" candidate_uuid: "841f8fa6250241d38d72dd5d8126f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87916da5b0e74a26add7c8341299fdc5" is_pre_election: true
I20260924 15:59:23.604034 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aa55a533014f7289ad50c6533663c7" candidate_uuid: "841f8fa6250241d38d72dd5d8126f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" is_pre_election: true
I20260924 15:59:23.604264 19544 raft_consensus.cc:2454] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841f8fa6250241d38d72dd5d8126f08c in term 0.
I20260924 15:59:23.604576 19405 raft_consensus.cc:2454] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841f8fa6250241d38d72dd5d8126f08c in term 0.
I20260924 15:59:23.605085 19192 leader_election.cc:302] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841f8fa6250241d38d72dd5d8126f08c, 87916da5b0e74a26add7c8341299fdc5; no voters:
I20260924 15:59:23.605504 19614 raft_consensus.cc:2781] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:23.605671 19614 raft_consensus.cc:482] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:23.605790 19614 raft_consensus.cc:3037] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:23.699110 19614 raft_consensus.cc:504] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } }
I20260924 15:59:23.699918 19614 leader_election.cc:288] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [CANDIDATE]: Term 1 election: Requested vote from peers 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639), ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853)
I20260924 15:59:23.700593 19544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aa55a533014f7289ad50c6533663c7" candidate_uuid: "841f8fa6250241d38d72dd5d8126f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87916da5b0e74a26add7c8341299fdc5"
I20260924 15:59:23.700631 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aa55a533014f7289ad50c6533663c7" candidate_uuid: "841f8fa6250241d38d72dd5d8126f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d"
I20260924 15:59:23.700853 19544 raft_consensus.cc:3037] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:23.700912 19405 raft_consensus.cc:3037] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:23.799911 19544 raft_consensus.cc:2454] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841f8fa6250241d38d72dd5d8126f08c in term 1.
I20260924 15:59:23.799952 19405 raft_consensus.cc:2454] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841f8fa6250241d38d72dd5d8126f08c in term 1.
I20260924 15:59:23.800741 19192 leader_election.cc:302] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841f8fa6250241d38d72dd5d8126f08c, 87916da5b0e74a26add7c8341299fdc5; no voters:
I20260924 15:59:23.801337 19614 raft_consensus.cc:2781] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:23.801983 19614 raft_consensus.cc:686] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 1 LEADER]: Becoming Leader. State: Replica: 841f8fa6250241d38d72dd5d8126f08c, State: Running, Role: LEADER
I20260924 15:59:23.802624 19614 consensus_queue.cc:237] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } }
I20260924 15:59:23.807664 19102 catalog_manager.cc:5495] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c reported cstate change: term changed from 0 to 1, leader changed from <none> to 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129). New cstate: current_term: 1 leader_uuid: "841f8fa6250241d38d72dd5d8126f08c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } health_report { overall_health: UNKNOWN } } }
Sep 24 15:59:23 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265561, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:23.882999 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.848485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.0.0.1:33838)
0924 15:59:23.849412 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:23.849428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:23.849553 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:23.849923 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:23.849926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:23.850777 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:23.851102 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.851111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.852191 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.852194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.853161 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:23.853171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.853851 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.853863 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:23.854074 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.862216 (+ 8142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:23.862236 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:23.864591 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:23.880321 (+ 15730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:23.880338 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:23.880348 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:23.880816 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:23.881185 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:23.881190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:23.881194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:23.881339 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:23.881871 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:23.881880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:23.882041 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 15:59:23.882208 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 15:59:23.882551 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":409,"threads_started":1}
I20260924 15:59:23.883060 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:23.848485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.16.233.190:36343)
0924 15:59:23.849220 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:23.849224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:23.849502 (+ 278us) server_negotiation.cc:407] Connection header received
0924 15:59:23.849651 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:23.849655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:23.849775 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:23.849895 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:23.851267 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.852076 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.853279 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:23.853667 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:23.853765 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:23.864827 (+ 11062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:23.864832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:23.864836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:23.864864 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:23.880150 (+ 15286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:23.880942 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:23.880949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:23.880952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:23.881074 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:23.881445 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:23.881452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:23.881455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:23.881716 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:23.881872 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:23.882345 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 15:59:23.882700 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":576,"thread_start_us":381,"threads_started":1}
I20260924 15:59:23.891323 19102 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:33838:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:59:23.892805 19102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:23.902693 19379 tablet_service.cc:1467] Processing CreateTablet for tablet fe926b4fd08f4dadb7ea4a11b42476ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52c2b0cc496f40ccb227a3cb13e82fc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:59:23.903055 19519 tablet_service.cc:1467] Processing CreateTablet for tablet fe926b4fd08f4dadb7ea4a11b42476ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52c2b0cc496f40ccb227a3cb13e82fc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:59:23.903841 19379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe926b4fd08f4dadb7ea4a11b42476ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:23.903906 19519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe926b4fd08f4dadb7ea4a11b42476ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:23.906908 19239 tablet_service.cc:1467] Processing CreateTablet for tablet fe926b4fd08f4dadb7ea4a11b42476ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52c2b0cc496f40ccb227a3cb13e82fc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:59:23.907891 19239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe926b4fd08f4dadb7ea4a11b42476ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:23.911934 19608 tablet_bootstrap.cc:492] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: Bootstrap starting.
I20260924 15:59:23.913285 19608 tablet_bootstrap.cc:654] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:23.914251 19610 tablet_bootstrap.cc:492] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: Bootstrap starting.
I20260924 15:59:23.916400 19610 tablet_bootstrap.cc:654] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:23.918340 19609 tablet_bootstrap.cc:492] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: Bootstrap starting.
I20260924 15:59:23.919919 19608 tablet_bootstrap.cc:492] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: No bootstrap required, opened a new log
I20260924 15:59:23.920033 19609 tablet_bootstrap.cc:654] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:23.920058 19608 ts_tablet_manager.cc:1397] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260924 15:59:23.920918 19608 raft_consensus.cc:348] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:23.921183 19608 raft_consensus.cc:374] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:23.921293 19608 raft_consensus.cc:729] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0e0695e2be4bd3b83cb79e1049fc4d, State: Initialized, Role: FOLLOWER
I20260924 15:59:23.921530 19608 consensus_queue.cc:260] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:23.921705 19610 tablet_bootstrap.cc:492] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: No bootstrap required, opened a new log
I20260924 15:59:23.921844 19610 ts_tablet_manager.cc:1397] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260924 15:59:23.922039 19615 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:23.922442 19615 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:23.922681 19610 raft_consensus.cc:348] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:23.922957 19610 raft_consensus.cc:374] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:23.923060 19610 raft_consensus.cc:729] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841f8fa6250241d38d72dd5d8126f08c, State: Initialized, Role: FOLLOWER
I20260924 15:59:23.923436 19608 ts_tablet_manager.cc:1428] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:59:23.923362 19610 consensus_queue.cc:260] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:23.923840 19614 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:23.924130 19614 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:23.925107 19610 ts_tablet_manager.cc:1428] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:23.926180 19609 tablet_bootstrap.cc:492] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: No bootstrap required, opened a new log
I20260924 15:59:23.926539 19609 ts_tablet_manager.cc:1397] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260924 15:59:23.927287 19609 raft_consensus.cc:348] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:23.927541 19609 raft_consensus.cc:374] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:23.927628 19609 raft_consensus.cc:729] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87916da5b0e74a26add7c8341299fdc5, State: Initialized, Role: FOLLOWER
I20260924 15:59:23.927872 19609 consensus_queue.cc:260] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:23.928608 19609 ts_tablet_manager.cc:1428] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:59:23.928653 19631 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:23.928957 19631 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:24.007539 19631 raft_consensus.cc:482] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:24.007833 19631 raft_consensus.cc:504] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.009837 19631 leader_election.cc:288] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167), ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853)
I20260924 15:59:24.021085 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.010061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39853 (local address 127.16.233.131:37281)
0924 15:59:24.013751 (+ 3690us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:24.013770 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:24.013876 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:24.014181 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:24.014185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:24.014250 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:24.014684 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.014697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.016263 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.016269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.018665 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.018684 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.020213 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.020219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.020410 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.020446 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 15:59:24.020537 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 15:59:24.020691 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3581,"mutex_wait_us":272,"spinlock_wait_cycles":3328,"thread_start_us":2915,"threads_started":1}
I20260924 15:59:24.021100 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.012652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:37281 (local address 127.16.233.130:39853)
0924 15:59:24.012910 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.012920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.013855 (+ 935us) server_negotiation.cc:407] Connection header received
0924 15:59:24.013978 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.013983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.014065 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.014154 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:24.014952 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.016098 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.018820 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.019863 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.020018 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.020095 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.020603 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.020800 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 15:59:24.021677 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.010061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:41167 (local address 127.16.233.131:39413)
0924 15:59:24.013478 (+ 3417us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:24.013504 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:24.013685 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:24.014105 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:24.014108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:24.014158 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:24.014543 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.014562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.016263 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.016269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.019160 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.019170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.020821 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.020828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.021023 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.021066 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:59:24.021166 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 15:59:24.021380 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3234,"mutex_wait_us":90,"thread_start_us":3347,"threads_started":1}
I20260924 15:59:24.021840 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.010316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:39413 (local address 127.16.233.129:41167)
0924 15:59:24.013291 (+ 2975us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.013299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.013688 (+ 389us) server_negotiation.cc:407] Connection header received
0924 15:59:24.013874 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.013881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.013967 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.014082 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:24.014741 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.016109 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.019340 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.020584 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.020734 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.020807 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.021387 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.021529 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2726,"thread_start_us":2567,"threads_started":1}
I20260924 15:59:24.022116 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe926b4fd08f4dadb7ea4a11b42476ba" candidate_uuid: "87916da5b0e74a26add7c8341299fdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" is_pre_election: true
I20260924 15:59:24.022423 19405 raft_consensus.cc:2454] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87916da5b0e74a26add7c8341299fdc5 in term 0.
I20260924 15:59:24.022918 19471 leader_election.cc:302] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87916da5b0e74a26add7c8341299fdc5, ea0e0695e2be4bd3b83cb79e1049fc4d; no voters:
I20260924 15:59:24.023406 19631 raft_consensus.cc:2781] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:24.023239 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe926b4fd08f4dadb7ea4a11b42476ba" candidate_uuid: "87916da5b0e74a26add7c8341299fdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841f8fa6250241d38d72dd5d8126f08c" is_pre_election: true
I20260924 15:59:24.023547 19631 raft_consensus.cc:482] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:24.023643 19631 raft_consensus.cc:3037] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:24.023664 19260 raft_consensus.cc:2454] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87916da5b0e74a26add7c8341299fdc5 in term 0.
I20260924 15:59:24.107473 19631 raft_consensus.cc:504] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.108227 19631 leader_election.cc:288] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [CANDIDATE]: Term 1 election: Requested vote from peers 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167), ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853)
I20260924 15:59:24.108850 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe926b4fd08f4dadb7ea4a11b42476ba" candidate_uuid: "87916da5b0e74a26add7c8341299fdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d"
I20260924 15:59:24.108884 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe926b4fd08f4dadb7ea4a11b42476ba" candidate_uuid: "87916da5b0e74a26add7c8341299fdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841f8fa6250241d38d72dd5d8126f08c"
I20260924 15:59:24.109110 19405 raft_consensus.cc:3037] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:24.109195 19260 raft_consensus.cc:3037] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:24.210296 19260 raft_consensus.cc:2454] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87916da5b0e74a26add7c8341299fdc5 in term 1.
I20260924 15:59:24.210244 19405 raft_consensus.cc:2454] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87916da5b0e74a26add7c8341299fdc5 in term 1.
I20260924 15:59:24.211162 19471 leader_election.cc:302] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87916da5b0e74a26add7c8341299fdc5, ea0e0695e2be4bd3b83cb79e1049fc4d; no voters:
I20260924 15:59:24.211539 19631 raft_consensus.cc:2781] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:24.211987 19631 raft_consensus.cc:686] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [term 1 LEADER]: Becoming Leader. State: Replica: 87916da5b0e74a26add7c8341299fdc5, State: Running, Role: LEADER
I20260924 15:59:24.212777 19631 consensus_queue.cc:237] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.216854 19635 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: TxnStatusTablet state changed. Reason: New leader 87916da5b0e74a26add7c8341299fdc5. Latest consensus state: current_term: 1 leader_uuid: "87916da5b0e74a26add7c8341299fdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:24.217165 19635 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:59:24.218533 19637 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:59:24.218827 19637 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:59:24.218636 19098 catalog_manager.cc:5495] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131). New cstate: current_term: 1 leader_uuid: "87916da5b0e74a26add7c8341299fdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } health_report { overall_health: HEALTHY } } }
I20260924 15:59:24.304010 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.295516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33639 (local address 127.0.0.1:60510)
0924 15:59:24.295854 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:24.295872 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:24.296011 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:24.297056 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:24.297062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:24.297077 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:59:24.297302 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.297314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.299123 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.299132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.300796 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.300815 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.301873 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.301880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.302100 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.302180 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.303366 (+ 1186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.303392 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 15:59:24.303504 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 15:59:24.303678 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260924 15:59:24.304807 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.295618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.16.233.131:33639)
0924 15:59:24.296615 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.296621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.296644 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:59:24.296737 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.296742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.296850 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.297000 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:24.297476 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.298962 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.300964 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.301696 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.301818 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.302480 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.303248 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.303382 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.304325 (+ 943us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.304422 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":644,"threads_started":1}
I20260924 15:59:24.314164 19260 raft_consensus.cc:1261] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Refusing update from remote peer 87916da5b0e74a26add7c8341299fdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:24.314186 19405 raft_consensus.cc:1261] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Refusing update from remote peer 87916da5b0e74a26add7c8341299fdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:24.315320 19635 consensus_queue.cc:1035] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:24.315675 19631 consensus_queue.cc:1035] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:24.325475 19615 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: TxnStatusTablet state changed. Reason: New leader 87916da5b0e74a26add7c8341299fdc5. Latest consensus state: current_term: 1 leader_uuid: "87916da5b0e74a26add7c8341299fdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:24.325884 19615 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:24.329474 19641 mvcc.cc:205] Tried to move back new op lower bound from 7332927751414177792 to 7332927751020519424. Current Snapshot: MvccSnapshot[applied={T|T < 7332927751414177792}]
I20260924 15:59:24.332827 19631 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87916da5b0e74a26add7c8341299fdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:24.333240 19631 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:59:24.336730 19614 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: TxnStatusTablet state changed. Reason: New leader 87916da5b0e74a26add7c8341299fdc5. Latest consensus state: current_term: 1 leader_uuid: "87916da5b0e74a26add7c8341299fdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:24.337046 19614 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:24.337059 19615 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87916da5b0e74a26add7c8341299fdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:24.337306 19615 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P ea0e0695e2be4bd3b83cb79e1049fc4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:24.340485 19635 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87916da5b0e74a26add7c8341299fdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:24.340790 19635 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P 87916da5b0e74a26add7c8341299fdc5: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:59:24.343019 19614 tablet_replica.cc:439] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87916da5b0e74a26add7c8341299fdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } } }
I20260924 15:59:24.343364 19614 tablet_replica.cc:442] T fe926b4fd08f4dadb7ea4a11b42476ba P 841f8fa6250241d38d72dd5d8126f08c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:59:24.353504 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.347112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.16.233.129:41167)
0924 15:59:24.347298 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.347306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.347719 (+ 413us) server_negotiation.cc:407] Connection header received
0924 15:59:24.347990 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.347994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.348066 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.348220 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:24.348656 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.349510 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.351233 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.351738 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.351876 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.352329 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.352871 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.353032 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.353112 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.353199 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 15:59:24.370970 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.364709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:41527 (local address 127.16.233.131:33639)
0924 15:59:24.364914 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.364921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.365463 (+ 542us) server_negotiation.cc:407] Connection header received
0924 15:59:24.365619 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.365625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.365733 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.365838 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:24.366284 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.367449 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.368878 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.369553 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.369688 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.370130 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.370267 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.370439 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.370582 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.370668 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 15:59:24.370998 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.364668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33639 (local address 127.16.233.129:41527)
0924 15:59:24.365361 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:24.365378 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:24.365495 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:24.365857 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:24.365860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:24.365869 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:59:24.366145 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.366152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.367603 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.367607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.368692 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.368709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.369732 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.369737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.369860 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.369919 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.370452 (+ 533us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.370462 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 15:59:24.370531 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:59:24.370640 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":371,"threads_started":1}
I20260924 15:59:24.381891 19405 raft_consensus.cc:1261] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Refusing update from remote peer 841f8fa6250241d38d72dd5d8126f08c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:24.382548 19544 raft_consensus.cc:1261] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Refusing update from remote peer 841f8fa6250241d38d72dd5d8126f08c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:24.382627 19656 consensus_queue.cc:1035] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:24.383276 19614 consensus_queue.cc:1035] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [LEADER]: Connected to new peer: Peer: permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:24.396802 19643 mvcc.cc:205] Tried to move back new op lower bound from 7332927751701712896 to 7332927749340680192. Current Snapshot: MvccSnapshot[applied={T|T < 7332927751701712896}]
I20260924 15:59:24.431274 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.424849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:56991 (local address 127.16.233.129:41167)
0924 15:59:24.425049 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.425055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.425900 (+ 845us) server_negotiation.cc:407] Connection header received
0924 15:59:24.426197 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.426203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.426293 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.426423 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:24.426986 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.427769 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.429374 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.429786 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.429923 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.430393 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.430561 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.430653 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.430863 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.430958 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 15:59:24.432008 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.425018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:41167 (local address 127.16.233.131:56991)
0924 15:59:24.425793 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:24.425814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:24.426037 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:24.426440 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:24.426445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:24.426457 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:59:24.426797 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.426807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.427900 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.427907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.429229 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.429243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.429963 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.429971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.430124 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.430185 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.430713 (+ 528us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.430725 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 15:59:24.431482 (+ 757us) client_negotiation.cc:240] Negotiation successful
0924 15:59:24.431657 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":546,"threads_started":1}
I20260924 15:59:24.475963 19102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:24.476307 19102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:24.481117 19102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d: Sending DeleteTablet for 3 replicas of tablet a1aa55a533014f7289ad50c6533663c7
I20260924 15:59:24.482486 19239 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aa55a533014f7289ad50c6533663c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:24 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:52608
I20260924 15:59:24.483239 19379 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aa55a533014f7289ad50c6533663c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:24 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:49482
I20260924 15:59:24.485766 19519 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aa55a533014f7289ad50c6533663c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:24 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:60494
I20260924 15:59:24.487962 19102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:59:24.490505 19102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:24.494829 19666 tablet_replica.cc:330] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: stopping tablet replica
I20260924 15:59:24.495534 19666 raft_consensus.cc:2229] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:59:24.496373 19666 raft_consensus.cc:2258] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:24.498711 19666 ts_tablet_manager.cc:1905] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:24.503731 19518 tablet_service.cc:1467] Processing CreateTablet for tablet 076ece305ce9491e8ee7f8845fb3849f (DEFAULT_TABLE table=test-table [id=776ca46ed2794fec902e9b7b41634125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:24.504338 19518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076ece305ce9491e8ee7f8845fb3849f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:24.504380 19239 tablet_service.cc:1467] Processing CreateTablet for tablet 076ece305ce9491e8ee7f8845fb3849f (DEFAULT_TABLE table=test-table [id=776ca46ed2794fec902e9b7b41634125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:24.504817 19239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076ece305ce9491e8ee7f8845fb3849f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:24.510805 19378 tablet_service.cc:1467] Processing CreateTablet for tablet 076ece305ce9491e8ee7f8845fb3849f (DEFAULT_TABLE table=test-table [id=776ca46ed2794fec902e9b7b41634125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:24.511394 19378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076ece305ce9491e8ee7f8845fb3849f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:24.515870 19666 ts_tablet_manager.cc:1918] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:59:24.517987 19666 log.cc:1196] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/a1aa55a533014f7289ad50c6533663c7
I20260924 15:59:24.518945 19666 ts_tablet_manager.cc:1939] T a1aa55a533014f7289ad50c6533663c7 P 841f8fa6250241d38d72dd5d8126f08c: Deleting consensus metadata
I20260924 15:59:24.522631 19667 tablet_replica.cc:330] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: stopping tablet replica
I20260924 15:59:24.522404 19668 tablet_replica.cc:330] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: stopping tablet replica
I20260924 15:59:24.524592 19087 catalog_manager.cc:4841] TS 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167): tablet a1aa55a533014f7289ad50c6533663c7 (table test-table [id=8c25a46b5e76443fb335d98efa3e7c73]) successfully deleted
I20260924 15:59:24.524951 19667 raft_consensus.cc:2229] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:24.525377 19669 tablet_bootstrap.cc:492] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: Bootstrap starting.
I20260924 15:59:24.525359 19668 raft_consensus.cc:2229] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:24.525712 19668 raft_consensus.cc:2258] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:24.526214 19667 raft_consensus.cc:2258] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:24.527802 19669 tablet_bootstrap.cc:654] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:24.528287 19668 ts_tablet_manager.cc:1905] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:24.533166 19667 ts_tablet_manager.cc:1905] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:24.533674 19671 tablet_bootstrap.cc:492] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: Bootstrap starting.
I20260924 15:59:24.535645 19671 tablet_bootstrap.cc:654] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:24.536689 19669 tablet_bootstrap.cc:492] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: No bootstrap required, opened a new log
I20260924 15:59:24.536957 19669 ts_tablet_manager.cc:1397] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260924 15:59:24.537707 19670 tablet_bootstrap.cc:492] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: Bootstrap starting.
I20260924 15:59:24.537863 19669 raft_consensus.cc:348] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.538240 19669 raft_consensus.cc:374] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:24.538416 19669 raft_consensus.cc:729] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841f8fa6250241d38d72dd5d8126f08c, State: Initialized, Role: FOLLOWER
I20260924 15:59:24.538769 19669 consensus_queue.cc:260] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.539548 19669 ts_tablet_manager.cc:1428] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:59:24.542052 19670 tablet_bootstrap.cc:654] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:24.550998 19671 tablet_bootstrap.cc:492] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: No bootstrap required, opened a new log
I20260924 15:59:24.551281 19671 ts_tablet_manager.cc:1397] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20260924 15:59:24.552306 19671 raft_consensus.cc:348] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.552692 19671 raft_consensus.cc:374] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:24.552855 19671 raft_consensus.cc:729] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0e0695e2be4bd3b83cb79e1049fc4d, State: Initialized, Role: FOLLOWER
I20260924 15:59:24.553267 19671 consensus_queue.cc:260] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.553970 19671 ts_tablet_manager.cc:1428] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:59:24.557811 19615 raft_consensus.cc:482] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:24.558107 19615 raft_consensus.cc:504] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.560424 19615 leader_election.cc:288] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167), 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639)
I20260924 15:59:24.591184 19668 ts_tablet_manager.cc:1918] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:59:24.591609 19668 log.cc:1196] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/a1aa55a533014f7289ad50c6533663c7
I20260924 15:59:24.598382 19668 ts_tablet_manager.cc:1939] T a1aa55a533014f7289ad50c6533663c7 P 87916da5b0e74a26add7c8341299fdc5: Deleting consensus metadata
I20260924 15:59:24.600244 19667 ts_tablet_manager.cc:1918] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:59:24.603056 19667 log.cc:1196] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/a1aa55a533014f7289ad50c6533663c7
I20260924 15:59:24.603986 19088 catalog_manager.cc:4841] TS 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639): tablet a1aa55a533014f7289ad50c6533663c7 (table test-table [id=8c25a46b5e76443fb335d98efa3e7c73]) successfully deleted
I20260924 15:59:24.605335 19667 ts_tablet_manager.cc:1939] T a1aa55a533014f7289ad50c6533663c7 P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting consensus metadata
I20260924 15:59:24.612396 19086 catalog_manager.cc:4841] TS ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853): tablet a1aa55a533014f7289ad50c6533663c7 (table test-table [id=8c25a46b5e76443fb335d98efa3e7c73]) successfully deleted
I20260924 15:59:24.617395 19670 tablet_bootstrap.cc:492] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: No bootstrap required, opened a new log
I20260924 15:59:24.617679 19670 ts_tablet_manager.cc:1397] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: Time spent bootstrapping tablet: real 0.080s user 0.040s sys 0.036s
I20260924 15:59:24.618695 19670 raft_consensus.cc:348] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.622079 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.562941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:34383 (local address 127.16.233.129:41167)
0924 15:59:24.563449 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.563460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.598990 (+ 35530us) server_negotiation.cc:407] Connection header received
0924 15:59:24.601979 (+ 2989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.601985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.602081 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.602143 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:24.607990 (+ 5847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.609116 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.619182 (+ 10066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.620670 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.620858 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.620945 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.621587 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.621754 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335}
I20260924 15:59:24.622891 19670 raft_consensus.cc:374] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:24.623086 19670 raft_consensus.cc:729] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87916da5b0e74a26add7c8341299fdc5, State: Initialized, Role: FOLLOWER
I20260924 15:59:24.623432 19670 consensus_queue.cc:260] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.624316 19670 ts_tablet_manager.cc:1428] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 15:59:24.619010 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.563194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:58637 (local address 127.16.233.131:33639)
0924 15:59:24.563674 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.563689 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.599527 (+ 35838us) server_negotiation.cc:407] Connection header received
0924 15:59:24.600904 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.600910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.601025 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.601090 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:24.603332 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.604573 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.616095 (+ 11522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.617553 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.617710 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.617804 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.618467 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.618643 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260924 15:59:24.629715 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.561626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:41167 (local address 127.16.233.130:34383)
0924 15:59:24.598857 (+ 37231us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:24.598882 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:24.600713 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:24.602375 (+ 1662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:24.602379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:24.602407 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:24.602795 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.602809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.609315 (+ 6506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.609321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.615434 (+ 6113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.615455 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.621176 (+ 5721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.621184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.621384 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.621419 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 15:59:24.622526 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0924 15:59:24.622773 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37048,"thread_start_us":38377,"threads_started":1}
I20260924 15:59:24.630473 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076ece305ce9491e8ee7f8845fb3849f" candidate_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841f8fa6250241d38d72dd5d8126f08c" is_pre_election: true
I20260924 15:59:24.630754 19260 raft_consensus.cc:2454] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea0e0695e2be4bd3b83cb79e1049fc4d in term 0.
I20260924 15:59:24.631377 19332 leader_election.cc:302] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841f8fa6250241d38d72dd5d8126f08c, ea0e0695e2be4bd3b83cb79e1049fc4d; no voters:
I20260924 15:59:24.631861 19615 raft_consensus.cc:2781] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:24.631994 19615 raft_consensus.cc:482] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:24.632090 19615 raft_consensus.cc:3037] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:24.635347 19615 raft_consensus.cc:504] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.636718 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076ece305ce9491e8ee7f8845fb3849f" candidate_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841f8fa6250241d38d72dd5d8126f08c"
I20260924 15:59:24.636976 19260 raft_consensus.cc:3037] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:24.642926 19260 raft_consensus.cc:2454] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0e0695e2be4bd3b83cb79e1049fc4d in term 1.
I20260924 15:59:24.643503 19332 leader_election.cc:302] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841f8fa6250241d38d72dd5d8126f08c, ea0e0695e2be4bd3b83cb79e1049fc4d; no voters:
I20260924 15:59:24.644104 19615 leader_election.cc:288] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [CANDIDATE]: Term 1 election: Requested vote from peers 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167), 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639)
I20260924 15:59:24.644322 19615 raft_consensus.cc:2781] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:24.645066 19615 raft_consensus.cc:686] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 LEADER]: Becoming Leader. State: Replica: ea0e0695e2be4bd3b83cb79e1049fc4d, State: Running, Role: LEADER
I20260924 15:59:24.645541 19615 consensus_queue.cc:237] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:24.619602 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.562118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33639 (local address 127.16.233.130:58637)
0924 15:59:24.599426 (+ 37308us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:24.599444 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:24.600524 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:24.601334 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:24.601340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:24.601374 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:24.601741 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.601754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.605559 (+ 3805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.605566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.607783 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.607801 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.618004 (+ 10203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.618013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.618201 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.618242 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 15:59:24.618730 (+ 488us) client_negotiation.cc:240] Negotiation successful
0924 15:59:24.619216 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37179,"thread_start_us":36495,"threads_started":1}
I20260924 15:59:24.651511 19544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076ece305ce9491e8ee7f8845fb3849f" candidate_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87916da5b0e74a26add7c8341299fdc5" is_pre_election: true
I20260924 15:59:24.651558 19543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076ece305ce9491e8ee7f8845fb3849f" candidate_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87916da5b0e74a26add7c8341299fdc5"
I20260924 15:59:24.651798 19544 raft_consensus.cc:2454] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea0e0695e2be4bd3b83cb79e1049fc4d in term 0.
I20260924 15:59:24.657768 19098 catalog_manager.cc:5495] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d reported cstate change: term changed from 0 to 1, leader changed from <none> to ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130). New cstate: current_term: 1 leader_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } health_report { overall_health: UNKNOWN } } }
I20260924 15:59:24.698136 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.688877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.16.233.130:39853)
0924 15:59:24.691074 (+ 2197us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.691080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.691104 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:59:24.691226 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.691232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.691338 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.691531 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:24.692039 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.693252 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.695212 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.695896 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.696027 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.696567 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.697387 (+ 820us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.697541 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.697662 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.697752 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2054,"thread_start_us":1906,"threads_started":1}
I20260924 15:59:24.718416 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.710910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:53569 (local address 127.16.233.131:33639)
0924 15:59:24.711355 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.711362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.711846 (+ 484us) server_negotiation.cc:407] Connection header received
0924 15:59:24.712062 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.712071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.712176 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.712287 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:24.712872 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.714042 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.716179 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.716877 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.717008 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.717569 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.717729 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.717829 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.717984 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.718072 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260924 15:59:24.718554 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.710936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33639 (local address 127.16.233.130:53569)
0924 15:59:24.711704 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:24.711742 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:24.711905 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:24.712319 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:24.712325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:24.712341 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:59:24.712703 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.712715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.714242 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.714249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.716022 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.716039 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.717072 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.717084 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.717284 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.717363 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.717850 (+ 487us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.717864 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:59:24.717949 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:59:24.718099 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":445,"threads_started":1}
I20260924 15:59:24.730468 19543 raft_consensus.cc:3037] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:24.730573 19260 raft_consensus.cc:1261] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Refusing update from remote peer ea0e0695e2be4bd3b83cb79e1049fc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:24.731213 19678 consensus_queue.cc:1035] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:24.734354 19543 raft_consensus.cc:1261] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Refusing update from remote peer ea0e0695e2be4bd3b83cb79e1049fc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:24.735174 19678 consensus_queue.cc:1035] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:24.751416 19642 mvcc.cc:205] Tried to move back new op lower bound from 7332927753130377216 to 7332927752801230848. Current Snapshot: MvccSnapshot[applied={T|T < 7332927753130377216}]
I20260924 15:59:24.799932 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.790942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39853 (local address 127.16.233.131:60101)
0924 15:59:24.791137 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:24.791157 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:24.791312 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:24.791873 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:24.791877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:24.791890 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:59:24.792092 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.792104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.793234 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.793241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.794864 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:24.794880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.798299 (+ 3419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.798325 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:24.798551 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.798607 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.799341 (+ 734us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.799353 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 15:59:24.799424 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:59:24.799599 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:59:24.800725 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:24.790488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:60101 (local address 127.16.233.130:39853)
0924 15:59:24.790705 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:24.790714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:24.791233 (+ 519us) server_negotiation.cc:407] Connection header received
0924 15:59:24.791463 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:24.791469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:24.791559 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:24.791666 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:24.792265 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.793087 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.795042 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:24.795737 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:24.795862 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:24.798802 (+ 2940us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:24.798936 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:24.799012 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:24.800358 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0924 15:59:24.800448 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 15:59:24.802235 19643 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 15:59:26.427233 19102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:26.427559 19102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:26.432422 19102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d: Sending DeleteTablet for 3 replicas of tablet 076ece305ce9491e8ee7f8845fb3849f
I20260924 15:59:26.433342 19239 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ece305ce9491e8ee7f8845fb3849f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:26 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:52608
I20260924 15:59:26.433724 19519 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ece305ce9491e8ee7f8845fb3849f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:26 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:60494
I20260924 15:59:26.434131 19709 tablet_replica.cc:330] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: stopping tablet replica
I20260924 15:59:26.434165 19378 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ece305ce9491e8ee7f8845fb3849f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:26 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:49482
I20260924 15:59:26.435010 19709 raft_consensus.cc:2229] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:26.435384 19709 raft_consensus.cc:2258] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:26.436833 19709 ts_tablet_manager.cc:1905] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:26.446434 19711 tablet_replica.cc:330] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: stopping tablet replica
I20260924 15:59:26.446723 19711 raft_consensus.cc:2229] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:59:26.447191 19709 ts_tablet_manager.cc:1918] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:59:26.447407 19711 raft_consensus.cc:2258] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:26.447496 19709 log.cc:1196] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/076ece305ce9491e8ee7f8845fb3849f
I20260924 15:59:26.448246 19710 tablet_replica.cc:330] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: stopping tablet replica
I20260924 15:59:26.448459 19709 ts_tablet_manager.cc:1939] T 076ece305ce9491e8ee7f8845fb3849f P 841f8fa6250241d38d72dd5d8126f08c: Deleting consensus metadata
I20260924 15:59:26.448621 19710 raft_consensus.cc:2229] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:26.448912 19711 ts_tablet_manager.cc:1905] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:26.449115 19710 raft_consensus.cc:2258] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:26.450030 19087 catalog_manager.cc:4841] TS 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167): tablet 076ece305ce9491e8ee7f8845fb3849f (table test-table [id=776ca46ed2794fec902e9b7b41634125]) successfully deleted
I20260924 15:59:26.450556 19710 ts_tablet_manager.cc:1905] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:26.455152 19711 ts_tablet_manager.cc:1918] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:59:26.455317 19711 log.cc:1196] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/076ece305ce9491e8ee7f8845fb3849f
I20260924 15:59:26.455698 19711 ts_tablet_manager.cc:1939] T 076ece305ce9491e8ee7f8845fb3849f P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting consensus metadata
I20260924 15:59:26.456741 19086 catalog_manager.cc:4841] TS ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853): tablet 076ece305ce9491e8ee7f8845fb3849f (table test-table [id=776ca46ed2794fec902e9b7b41634125]) successfully deleted
I20260924 15:59:26.457393 19710 ts_tablet_manager.cc:1918] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:59:26.457556 19710 log.cc:1196] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/076ece305ce9491e8ee7f8845fb3849f
I20260924 15:59:26.458000 19710 ts_tablet_manager.cc:1939] T 076ece305ce9491e8ee7f8845fb3849f P 87916da5b0e74a26add7c8341299fdc5: Deleting consensus metadata
I20260924 15:59:26.459249 19088 catalog_manager.cc:4841] TS 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639): tablet 076ece305ce9491e8ee7f8845fb3849f (table test-table [id=776ca46ed2794fec902e9b7b41634125]) successfully deleted
Sep 24 15:59:26 dist-test-slave-xn4h krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:59:26 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:26.506487 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:26.483518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.16.233.190:36343)
0924 15:59:26.484145 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:26.484151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:26.484178 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:59:26.484355 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:26.484359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:26.484429 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:26.484531 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:26.485995 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.486870 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.488092 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.488483 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.488585 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:26.499993 (+ 11408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:26.500000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:26.500004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:26.500053 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:26.503684 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:26.504690 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:26.504695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:26.504697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:26.504837 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:26.505223 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:26.505226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:26.505228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:26.505501 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:26.505679 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:26.505955 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 15:59:26.506124 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":351,"threads_started":1}
I20260924 15:59:26.510679 19102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:59:26.511381 19102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:26.518350 19239 tablet_service.cc:1467] Processing CreateTablet for tablet c70256a90b8b4ec880585abbd66b5d93 (DEFAULT_TABLE table=test-table [id=8c93e63a29ac4b58adbbc2e98ac0d2cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:26.518714 19378 tablet_service.cc:1467] Processing CreateTablet for tablet c70256a90b8b4ec880585abbd66b5d93 (DEFAULT_TABLE table=test-table [id=8c93e63a29ac4b58adbbc2e98ac0d2cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:26.518862 19239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70256a90b8b4ec880585abbd66b5d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:26.518890 19519 tablet_service.cc:1467] Processing CreateTablet for tablet c70256a90b8b4ec880585abbd66b5d93 (DEFAULT_TABLE table=test-table [id=8c93e63a29ac4b58adbbc2e98ac0d2cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:26.519063 19378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70256a90b8b4ec880585abbd66b5d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:26.519282 19519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70256a90b8b4ec880585abbd66b5d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:26.528838 19722 tablet_bootstrap.cc:492] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: Bootstrap starting.
I20260924 15:59:26.530638 19721 tablet_bootstrap.cc:492] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: Bootstrap starting.
I20260924 15:59:26.531965 19721 tablet_bootstrap.cc:654] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:26.532045 19722 tablet_bootstrap.cc:654] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:26.532786 19720 tablet_bootstrap.cc:492] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: Bootstrap starting.
I20260924 15:59:26.535265 19720 tablet_bootstrap.cc:654] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:26.535382 19721 tablet_bootstrap.cc:492] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: No bootstrap required, opened a new log
I20260924 15:59:26.535392 19722 tablet_bootstrap.cc:492] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: No bootstrap required, opened a new log
I20260924 15:59:26.535547 19722 ts_tablet_manager.cc:1397] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260924 15:59:26.535545 19721 ts_tablet_manager.cc:1397] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 15:59:26.536480 19722 raft_consensus.cc:348] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.536495 19721 raft_consensus.cc:348] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.536739 19721 raft_consensus.cc:374] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:26.536847 19721 raft_consensus.cc:729] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841f8fa6250241d38d72dd5d8126f08c, State: Initialized, Role: FOLLOWER
I20260924 15:59:26.536868 19722 raft_consensus.cc:374] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:26.536965 19722 raft_consensus.cc:729] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87916da5b0e74a26add7c8341299fdc5, State: Initialized, Role: FOLLOWER
I20260924 15:59:26.537230 19721 consensus_queue.cc:260] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.537261 19722 consensus_queue.cc:260] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.537933 19722 ts_tablet_manager.cc:1428] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 15:59:26.538682 19721 ts_tablet_manager.cc:1428] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:26.540992 19720 tablet_bootstrap.cc:492] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: No bootstrap required, opened a new log
I20260924 15:59:26.541117 19720 ts_tablet_manager.cc:1397] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260924 15:59:26.541863 19720 raft_consensus.cc:348] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.542140 19720 raft_consensus.cc:374] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:26.542253 19720 raft_consensus.cc:729] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0e0695e2be4bd3b83cb79e1049fc4d, State: Initialized, Role: FOLLOWER
I20260924 15:59:26.542558 19720 consensus_queue.cc:260] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.543182 19720 ts_tablet_manager.cc:1428] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:59:26.688242 19726 raft_consensus.cc:482] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:26.688527 19726 raft_consensus.cc:504] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.689414 19726 leader_election.cc:288] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853), 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639)
I20260924 15:59:26.690158 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70256a90b8b4ec880585abbd66b5d93" candidate_uuid: "841f8fa6250241d38d72dd5d8126f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" is_pre_election: true
I20260924 15:59:26.690112 19544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70256a90b8b4ec880585abbd66b5d93" candidate_uuid: "841f8fa6250241d38d72dd5d8126f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87916da5b0e74a26add7c8341299fdc5" is_pre_election: true
I20260924 15:59:26.690430 19544 raft_consensus.cc:2454] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841f8fa6250241d38d72dd5d8126f08c in term 0.
I20260924 15:59:26.690516 19405 raft_consensus.cc:2454] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841f8fa6250241d38d72dd5d8126f08c in term 0.
I20260924 15:59:26.691032 19192 leader_election.cc:302] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841f8fa6250241d38d72dd5d8126f08c, 87916da5b0e74a26add7c8341299fdc5; no voters:
I20260924 15:59:26.691350 19726 raft_consensus.cc:2781] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:26.691470 19726 raft_consensus.cc:482] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:26.691560 19726 raft_consensus.cc:3037] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:26.693580 19726 raft_consensus.cc:504] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.694131 19726 leader_election.cc:288] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [CANDIDATE]: Term 1 election: Requested vote from peers ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853), 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639)
I20260924 15:59:26.694728 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70256a90b8b4ec880585abbd66b5d93" candidate_uuid: "841f8fa6250241d38d72dd5d8126f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d"
I20260924 15:59:26.694753 19544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70256a90b8b4ec880585abbd66b5d93" candidate_uuid: "841f8fa6250241d38d72dd5d8126f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87916da5b0e74a26add7c8341299fdc5"
I20260924 15:59:26.694931 19405 raft_consensus.cc:3037] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:26.694994 19544 raft_consensus.cc:3037] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:26.696954 19405 raft_consensus.cc:2454] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841f8fa6250241d38d72dd5d8126f08c in term 1.
I20260924 15:59:26.697360 19190 leader_election.cc:302] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841f8fa6250241d38d72dd5d8126f08c, ea0e0695e2be4bd3b83cb79e1049fc4d; no voters:
I20260924 15:59:26.697587 19544 raft_consensus.cc:2454] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841f8fa6250241d38d72dd5d8126f08c in term 1.
I20260924 15:59:26.697757 19726 raft_consensus.cc:2781] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:26.698362 19726 raft_consensus.cc:686] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 1 LEADER]: Becoming Leader. State: Replica: 841f8fa6250241d38d72dd5d8126f08c, State: Running, Role: LEADER
I20260924 15:59:26.698699 19726 consensus_queue.cc:237] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.701170 19098 catalog_manager.cc:5495] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c reported cstate change: term changed from 0 to 1, leader changed from <none> to 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129). New cstate: current_term: 1 leader_uuid: "841f8fa6250241d38d72dd5d8126f08c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } health_report { overall_health: UNKNOWN } } }
I20260924 15:59:26.735426 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:26.729903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.16.233.129:41167)
0924 15:59:26.730682 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:26.730687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:26.730706 (+ 19us) server_negotiation.cc:407] Connection header received
0924 15:59:26.730754 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:26.730757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:26.730813 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:26.730891 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:26.731221 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.732181 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.733352 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.733992 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.734087 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:26.734701 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:26.734838 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:26.734917 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:26.735010 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 15:59:26.735068 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":542,"threads_started":1}
I20260924 15:59:26.744455 19544 raft_consensus.cc:1261] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Refusing update from remote peer 841f8fa6250241d38d72dd5d8126f08c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:26.744536 19405 raft_consensus.cc:1261] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Refusing update from remote peer 841f8fa6250241d38d72dd5d8126f08c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:26.745059 19727 consensus_queue.cc:1035] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [LEADER]: Connected to new peer: Peer: permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:26.745409 19726 consensus_queue.cc:1035] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:26.793485 19102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:26.793764 19102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:26.796932 19102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d: Sending DeleteTablet for 3 replicas of tablet c70256a90b8b4ec880585abbd66b5d93
I20260924 15:59:26.797919 19239 tablet_service.cc:1514] Processing DeleteTablet for tablet c70256a90b8b4ec880585abbd66b5d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:26 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:52608
I20260924 15:59:26.798131 19519 tablet_service.cc:1514] Processing DeleteTablet for tablet c70256a90b8b4ec880585abbd66b5d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:26 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:60494
I20260924 15:59:26.798504 19710 tablet_replica.cc:330] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: stopping tablet replica
I20260924 15:59:26.798583 19709 tablet_replica.cc:330] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: stopping tablet replica
I20260924 15:59:26.798754 19710 raft_consensus.cc:2229] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:26.798892 19709 raft_consensus.cc:2229] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:59:26.799009 19710 raft_consensus.cc:2258] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:26.799379 19709 raft_consensus.cc:2258] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:26.799362 19102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 15:59:26.799731 19378 tablet_service.cc:1514] Processing DeleteTablet for tablet c70256a90b8b4ec880585abbd66b5d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:26 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:49482
W20260924 15:59:26.800046 19102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:26.800177 19711 tablet_replica.cc:330] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: stopping tablet replica
I20260924 15:59:26.800175 19710 ts_tablet_manager.cc:1905] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:26.800391 19711 raft_consensus.cc:2229] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:26.800593 19711 raft_consensus.cc:2258] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:26.802810 19709 ts_tablet_manager.cc:1905] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:26.808301 19378 tablet_service.cc:1467] Processing CreateTablet for tablet ced7cc547736455e8e45717bbb0e6464 (DEFAULT_TABLE table=test-table [id=9645e63dd43b42a9b0f4c1cd45bcfe85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:26.808746 19378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ced7cc547736455e8e45717bbb0e6464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:26.808929 19519 tablet_service.cc:1467] Processing CreateTablet for tablet ced7cc547736455e8e45717bbb0e6464 (DEFAULT_TABLE table=test-table [id=9645e63dd43b42a9b0f4c1cd45bcfe85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:26.809389 19519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ced7cc547736455e8e45717bbb0e6464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:26.811306 19710 ts_tablet_manager.cc:1918] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:59:26.811496 19710 log.cc:1196] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/c70256a90b8b4ec880585abbd66b5d93
I20260924 15:59:26.811955 19710 ts_tablet_manager.cc:1939] T c70256a90b8b4ec880585abbd66b5d93 P 87916da5b0e74a26add7c8341299fdc5: Deleting consensus metadata
I20260924 15:59:26.812801 19239 tablet_service.cc:1467] Processing CreateTablet for tablet ced7cc547736455e8e45717bbb0e6464 (DEFAULT_TABLE table=test-table [id=9645e63dd43b42a9b0f4c1cd45bcfe85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:26.813244 19239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ced7cc547736455e8e45717bbb0e6464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:26.814486 19088 catalog_manager.cc:4841] TS 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639): tablet c70256a90b8b4ec880585abbd66b5d93 (table test-table [id=8c93e63a29ac4b58adbbc2e98ac0d2cc]) successfully deleted
I20260924 15:59:26.816126 19722 tablet_bootstrap.cc:492] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: Bootstrap starting.
I20260924 15:59:26.819010 19722 tablet_bootstrap.cc:654] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:26.821394 19722 tablet_bootstrap.cc:492] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: No bootstrap required, opened a new log
I20260924 15:59:26.821604 19722 ts_tablet_manager.cc:1397] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 15:59:26.822518 19722 raft_consensus.cc:348] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.822795 19722 raft_consensus.cc:374] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:26.822925 19722 raft_consensus.cc:729] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87916da5b0e74a26add7c8341299fdc5, State: Initialized, Role: FOLLOWER
I20260924 15:59:26.823269 19722 consensus_queue.cc:260] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.823995 19722 ts_tablet_manager.cc:1428] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:59:26.825223 19720 tablet_bootstrap.cc:492] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: Bootstrap starting.
I20260924 15:59:26.825551 19709 ts_tablet_manager.cc:1918] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:59:26.825743 19709 log.cc:1196] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/c70256a90b8b4ec880585abbd66b5d93
I20260924 15:59:26.826205 19709 ts_tablet_manager.cc:1939] T c70256a90b8b4ec880585abbd66b5d93 P 841f8fa6250241d38d72dd5d8126f08c: Deleting consensus metadata
I20260924 15:59:26.827499 19087 catalog_manager.cc:4841] TS 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167): tablet c70256a90b8b4ec880585abbd66b5d93 (table test-table [id=8c93e63a29ac4b58adbbc2e98ac0d2cc]) successfully deleted
I20260924 15:59:26.827755 19720 tablet_bootstrap.cc:654] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:26.828249 19721 tablet_bootstrap.cc:492] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: Bootstrap starting.
I20260924 15:59:26.829869 19711 ts_tablet_manager.cc:1905] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:26.831240 19721 tablet_bootstrap.cc:654] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:26.839250 19711 ts_tablet_manager.cc:1918] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:59:26.839445 19711 log.cc:1196] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/c70256a90b8b4ec880585abbd66b5d93
I20260924 15:59:26.839936 19711 ts_tablet_manager.cc:1939] T c70256a90b8b4ec880585abbd66b5d93 P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting consensus metadata
I20260924 15:59:26.840005 19720 tablet_bootstrap.cc:492] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: No bootstrap required, opened a new log
I20260924 15:59:26.840183 19720 ts_tablet_manager.cc:1397] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.013s
I20260924 15:59:26.841158 19086 catalog_manager.cc:4841] TS ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853): tablet c70256a90b8b4ec880585abbd66b5d93 (table test-table [id=8c93e63a29ac4b58adbbc2e98ac0d2cc]) successfully deleted
I20260924 15:59:26.841135 19720 raft_consensus.cc:348] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.841588 19720 raft_consensus.cc:374] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:26.841822 19720 raft_consensus.cc:729] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0e0695e2be4bd3b83cb79e1049fc4d, State: Initialized, Role: FOLLOWER
I20260924 15:59:26.842180 19720 consensus_queue.cc:260] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.842759 19721 tablet_bootstrap.cc:492] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: No bootstrap required, opened a new log
I20260924 15:59:26.842960 19721 ts_tablet_manager.cc:1397] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260924 15:59:26.842983 19720 ts_tablet_manager.cc:1428] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:59:26.843710 19721 raft_consensus.cc:348] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.843984 19721 raft_consensus.cc:374] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:26.844074 19721 raft_consensus.cc:729] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841f8fa6250241d38d72dd5d8126f08c, State: Initialized, Role: FOLLOWER
I20260924 15:59:26.844354 19721 consensus_queue.cc:260] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.844936 19721 ts_tablet_manager.cc:1428] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 15:59:26.854586 19649 raft_consensus.cc:482] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:26.854782 19649 raft_consensus.cc:504] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.855427 19649 leader_election.cc:288] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167), ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853)
I20260924 15:59:26.856068 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced7cc547736455e8e45717bbb0e6464" candidate_uuid: "87916da5b0e74a26add7c8341299fdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841f8fa6250241d38d72dd5d8126f08c" is_pre_election: true
I20260924 15:59:26.856068 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced7cc547736455e8e45717bbb0e6464" candidate_uuid: "87916da5b0e74a26add7c8341299fdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" is_pre_election: true
I20260924 15:59:26.856328 19260 raft_consensus.cc:2454] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87916da5b0e74a26add7c8341299fdc5 in term 0.
I20260924 15:59:26.856369 19405 raft_consensus.cc:2454] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87916da5b0e74a26add7c8341299fdc5 in term 0.
I20260924 15:59:26.856820 19471 leader_election.cc:302] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87916da5b0e74a26add7c8341299fdc5, ea0e0695e2be4bd3b83cb79e1049fc4d; no voters:
I20260924 15:59:26.857146 19649 raft_consensus.cc:2781] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:26.857326 19649 raft_consensus.cc:482] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:26.857515 19649 raft_consensus.cc:3037] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:26.860148 19649 raft_consensus.cc:504] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.860678 19649 leader_election.cc:288] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [CANDIDATE]: Term 1 election: Requested vote from peers 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167), ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853)
I20260924 15:59:26.861219 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced7cc547736455e8e45717bbb0e6464" candidate_uuid: "87916da5b0e74a26add7c8341299fdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841f8fa6250241d38d72dd5d8126f08c"
I20260924 15:59:26.861212 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced7cc547736455e8e45717bbb0e6464" candidate_uuid: "87916da5b0e74a26add7c8341299fdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d"
I20260924 15:59:26.861439 19260 raft_consensus.cc:3037] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:26.861451 19405 raft_consensus.cc:3037] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:26.863641 19405 raft_consensus.cc:2454] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87916da5b0e74a26add7c8341299fdc5 in term 1.
I20260924 15:59:26.863646 19260 raft_consensus.cc:2454] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87916da5b0e74a26add7c8341299fdc5 in term 1.
I20260924 15:59:26.864096 19471 leader_election.cc:302] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87916da5b0e74a26add7c8341299fdc5, ea0e0695e2be4bd3b83cb79e1049fc4d; no voters:
I20260924 15:59:26.864430 19649 raft_consensus.cc:2781] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:26.864742 19649 raft_consensus.cc:686] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 1 LEADER]: Becoming Leader. State: Replica: 87916da5b0e74a26add7c8341299fdc5, State: Running, Role: LEADER
I20260924 15:59:26.865052 19649 consensus_queue.cc:237] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } }
I20260924 15:59:26.867976 19102 catalog_manager.cc:5495] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131). New cstate: current_term: 1 leader_uuid: "87916da5b0e74a26add7c8341299fdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87916da5b0e74a26add7c8341299fdc5" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33639 } health_report { overall_health: HEALTHY } } }
I20260924 15:59:26.899978 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:26.894001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.16.233.131:33639)
0924 15:59:26.895171 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:26.895177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:26.895212 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:59:26.895264 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:26.895267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:26.895328 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:26.895410 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:26.895779 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.896903 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.898161 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.898790 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.898890 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:26.899297 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:26.899407 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:26.899474 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:26.899648 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 15:59:26.899718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044,"thread_start_us":805,"threads_started":1}
I20260924 15:59:26.915664 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:26.909432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:46927 (local address 127.16.233.131:33639)
0924 15:59:26.909629 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:26.909638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:26.910697 (+ 1059us) server_negotiation.cc:407] Connection header received
0924 15:59:26.910827 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:26.910830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:26.910889 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:26.910961 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:26.911644 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.912378 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.914180 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.914686 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.914793 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:26.915105 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:26.915198 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:26.915285 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:26.915368 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 15:59:26.915427 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":41}
I20260924 15:59:26.915762 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:26.909483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33639 (local address 127.16.233.131:46927)
0924 15:59:26.910599 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:26.910619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:26.910742 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:26.910986 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:26.910989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:26.910998 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:59:26.911494 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:26.911503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.912514 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.912517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:26.913946 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:26.913958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:26.914828 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:26.914832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:26.914944 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:26.914979 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:59:26.915289 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:59:26.915297 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 15:59:26.915354 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:59:26.915450 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":807,"threads_started":1}
I20260924 15:59:26.923841 19405 raft_consensus.cc:1261] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Refusing update from remote peer 87916da5b0e74a26add7c8341299fdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:26.924027 19260 raft_consensus.cc:1261] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Refusing update from remote peer 87916da5b0e74a26add7c8341299fdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:26.924491 19691 consensus_queue.cc:1035] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0e0695e2be4bd3b83cb79e1049fc4d" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:26.924973 19729 consensus_queue.cc:1035] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "841f8fa6250241d38d72dd5d8126f08c" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:59:26.946379 19735 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 15:59:28.537940 19100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:28.538185 19100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:28.542559 19100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b487575036240b6a285d43f3f42440d: Sending DeleteTablet for 3 replicas of tablet ced7cc547736455e8e45717bbb0e6464
I20260924 15:59:28.543500 19239 tablet_service.cc:1514] Processing DeleteTablet for tablet ced7cc547736455e8e45717bbb0e6464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:28 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:52608
I20260924 15:59:28.543627 19378 tablet_service.cc:1514] Processing DeleteTablet for tablet ced7cc547736455e8e45717bbb0e6464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:28 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:49482
I20260924 15:59:28.543902 19518 tablet_service.cc:1514] Processing DeleteTablet for tablet ced7cc547736455e8e45717bbb0e6464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:28 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:60494
I20260924 15:59:28.544524 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19173
I20260924 15:59:28.556529 19771 tablet_replica.cc:330] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: stopping tablet replica
I20260924 15:59:28.556838 19770 tablet_replica.cc:330] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: stopping tablet replica
I20260924 15:59:28.557097 19771 raft_consensus.cc:2229] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:59:28.557142 19770 raft_consensus.cc:2229] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:28.557490 19770 raft_consensus.cc:2258] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:28.558604 19771 raft_consensus.cc:2258] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:28.559139 19770 ts_tablet_manager.cc:1905] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:28.560247 19771 ts_tablet_manager.cc:1905] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:28.567582 19771 ts_tablet_manager.cc:1918] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:59:28.567864 19771 log.cc:1196] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/ced7cc547736455e8e45717bbb0e6464
I20260924 15:59:28.568491 19771 ts_tablet_manager.cc:1939] T ced7cc547736455e8e45717bbb0e6464 P 87916da5b0e74a26add7c8341299fdc5: Deleting consensus metadata
I20260924 15:59:28.568825 19770 ts_tablet_manager.cc:1918] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:59:28.569018 19770 log.cc:1196] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/ced7cc547736455e8e45717bbb0e6464
I20260924 15:59:28.569643 19770 ts_tablet_manager.cc:1939] T ced7cc547736455e8e45717bbb0e6464 P ea0e0695e2be4bd3b83cb79e1049fc4d: Deleting consensus metadata
Sep 24 15:59:28 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:28.570164 19088 catalog_manager.cc:4841] TS 87916da5b0e74a26add7c8341299fdc5 (127.16.233.131:33639): tablet ced7cc547736455e8e45717bbb0e6464 (table test-table [id=9645e63dd43b42a9b0f4c1cd45bcfe85]) successfully deleted
I20260924 15:59:28.570673 19769 tablet_replica.cc:330] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: stopping tablet replica
I20260924 15:59:28.570920 19769 raft_consensus.cc:2229] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:28.570945 19086 catalog_manager.cc:4841] TS ea0e0695e2be4bd3b83cb79e1049fc4d (127.16.233.130:39853): tablet ced7cc547736455e8e45717bbb0e6464 (table test-table [id=9645e63dd43b42a9b0f4c1cd45bcfe85]) successfully deleted
I20260924 15:59:28.571200 19769 raft_consensus.cc:2258] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:28.572227 19769 ts_tablet_manager.cc:1905] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:28.578112 19769 ts_tablet_manager.cc:1918] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:59:28.578322 19769 log.cc:1196] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/ced7cc547736455e8e45717bbb0e6464
I20260924 15:59:28.578655 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:28.547500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.16.233.129:41167)
0924 15:59:28.552489 (+ 4989us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:28.552499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:28.558896 (+ 6397us) server_negotiation.cc:407] Connection header received
0924 15:59:28.559476 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:28.559483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:28.559590 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:28.559684 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:28.561760 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:28.562977 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:28.564805 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:28.565463 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:28.565570 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:28.571718 (+ 6148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:28.571747 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:28.571783 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:28.571848 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:28.575209 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:28.576168 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:28.576177 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:28.576181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:28.576358 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:28.576803 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:28.576807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:28.576809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:28.577193 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:28.577470 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:28.577918 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 15:59:28.578138 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4799,"thread_start_us":4623,"threads_started":1}
I20260924 15:59:28.578910 19769 ts_tablet_manager.cc:1939] T ced7cc547736455e8e45717bbb0e6464 P 841f8fa6250241d38d72dd5d8126f08c: Deleting consensus metadata
W20260924 15:59:28.579458 19305 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40970
I20260924 15:59:28.579948 19087 catalog_manager.cc:4841] TS 841f8fa6250241d38d72dd5d8126f08c (127.16.233.129:41167): tablet ced7cc547736455e8e45717bbb0e6464 (table test-table [id=9645e63dd43b42a9b0f4c1cd45bcfe85]) successfully deleted
W20260924 15:59:28.580111 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19173: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 15:59:28.580369 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 19173
I20260924 15:59:28.603787 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19314
Sep 24 15:59:28 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 15:59:28.619798 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:28.604628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.16.233.130:39853)
0924 15:59:28.605101 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:28.605105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:28.605121 (+ 16us) server_negotiation.cc:407] Connection header received
0924 15:59:28.605176 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:28.605181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:28.605245 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:28.605345 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:28.606588 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:28.607317 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:28.608618 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:28.608992 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:28.609082 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:28.614073 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:28.614097 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:28.614122 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:28.614183 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:28.617103 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:28.617793 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:28.617800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:28.617803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:28.617953 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:28.618445 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:28.618452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:28.618455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:28.618848 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:28.619074 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:28.619344 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 15:59:28.619524 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":237,"threads_started":1}
W20260924 15:59:28.620540 19445 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47828
W20260924 15:59:28.621286 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19314: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 15:59:28.621479 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 19314
I20260924 15:59:28.653510 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19454
Sep 24 15:59:28 dist-test-slave-xn4h krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 15:59:28.670768 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:28.654359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.16.233.131:33639)
0924 15:59:28.654860 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:28.654864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:28.654881 (+ 17us) server_negotiation.cc:407] Connection header received
0924 15:59:28.654936 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:28.654940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:28.654997 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:28.655087 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:28.656708 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:28.657468 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:28.659438 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:28.659840 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:28.659934 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:28.664710 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:28.664728 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:28.664755 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:28.664799 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:28.667725 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:28.668520 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:28.668528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:28.668530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:28.668683 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:28.669085 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:28.669089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:28.669091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:28.669492 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:28.669808 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:28.670129 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 15:59:28.670410 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":281,"threads_started":1}
W20260924 15:59:28.671468 19584 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39594
W20260924 15:59:28.672092 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19454: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 15:59:28.672282 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 19454
I20260924 15:59:28.698935 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19069
I20260924 15:59:28.700223 19142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:28.817688 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 19069
2026-09-24T15:59:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8912 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:28 dist-test-slave-xn4h krb5kdc[19779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:28 dist-test-slave-xn4h krb5kdc[19779](info): set up 2 sockets
Sep 24 15:59:28 dist-test-slave-xn4h krb5kdc[19779](info): commencing operation
krb5kdc: starting...
W20260924 15:59:29.919975 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:30 dist-test-slave-xn4h krb5kdc[19779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:30.122769 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:35421
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:35421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:30.453308 19792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:30.453637 19792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:30.453728 19792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:30.461181 19792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:30.461310 19792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:30.461357 19792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:30.461413 19792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:30.461462 19792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:30.471730 19792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:35421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:35421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:30.473073 19792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:30.474409 19792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:30.486761 19797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:30.486727 19798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:30.486918 19800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:30.487789 19792 server_base.cc:1034] running on GCE node
I20260924 15:59:30.488598 19792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:30.490084 19792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:30.491290 19792 hybrid_clock.cc:648] HybridClock initialized: now 1790265570491223 us; error 52 us; skew 500 ppm
I20260924 15:59:30.491664 19792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:30 dist-test-slave-xn4h krb5kdc[19779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:30.494769 19792 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:30.496218 19792 webserver.cc:460] Webserver started at http://127.16.233.190:44789/ using document root <none> and password file <none>
I20260924 15:59:30.496903 19792 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:30.497001 19792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:30.497259 19792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:30.500288 19792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "eb8f0e0ff6a04870a8ad685c4213a90e"
format_stamp: "Formatted at 2026-09-24 15:59:30 on dist-test-slave-xn4h"
server_key: "ef3b626b2d5a47ba5d7c55e74609de6d"
server_key_iv: "35837a1d06c010a4607137df4f9b5518"
server_key_version: "encryptionkey@0"
I20260924 15:59:30.501029 19792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "eb8f0e0ff6a04870a8ad685c4213a90e"
format_stamp: "Formatted at 2026-09-24 15:59:30 on dist-test-slave-xn4h"
server_key: "ef3b626b2d5a47ba5d7c55e74609de6d"
server_key_iv: "35837a1d06c010a4607137df4f9b5518"
server_key_version: "encryptionkey@0"
I20260924 15:59:30.506594 19792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 15:59:30.510202 19807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:30.511706 19792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:30.511914 19792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "eb8f0e0ff6a04870a8ad685c4213a90e"
format_stamp: "Formatted at 2026-09-24 15:59:30 on dist-test-slave-xn4h"
server_key: "ef3b626b2d5a47ba5d7c55e74609de6d"
server_key_iv: "35837a1d06c010a4607137df4f9b5518"
server_key_version: "encryptionkey@0"
I20260924 15:59:30.512135 19792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:30.531571 19792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:30.536221 19792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:30.536557 19792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:30.556339 19792 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:35421
I20260924 15:59:30.556378 19859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:35421 every 8 connection(s)
I20260924 15:59:30.557812 19792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:30.561767 19860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:30.562471 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 19792
I20260924 15:59:30.562717 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:30.563344 17318 external_mini_cluster.cc:1428] Setting key c511484107706d9077567fcd6c23f447
I20260924 15:59:30.576538 19860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e: Bootstrap starting.
Sep 24 15:59:30 dist-test-slave-xn4h krb5kdc[19779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:30.580670 19860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:30.581848 19860 log.cc:824] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:30.584841 19860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e: No bootstrap required, opened a new log
I20260924 15:59:30.587674 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:30.565494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.16.233.190:35421)
0924 15:59:30.566385 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:30.566398 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:30.566445 (+ 47us) server_negotiation.cc:407] Connection header received
0924 15:59:30.568151 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:30.568188 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:30.568621 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:30.568981 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:30.570836 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:30.572199 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:30.574070 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:30.574847 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:30.574987 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:30.579828 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:30.579844 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:30.579864 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:30.579924 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:30.583374 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:30.584124 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:30.584133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:30.584139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:30.584307 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:30.584706 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:30.584712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:30.584715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:30.585329 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:30.585658 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:30.586098 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 15:59:30.586594 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":382,"threads_started":1}
I20260924 15:59:30.593017 19860 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8f0e0ff6a04870a8ad685c4213a90e" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35421 } }
I20260924 15:59:30.593426 19860 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:30.593537 19860 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8f0e0ff6a04870a8ad685c4213a90e, State: Initialized, Role: FOLLOWER
I20260924 15:59:30.594141 19860 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8f0e0ff6a04870a8ad685c4213a90e" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35421 } }
I20260924 15:59:30.594419 19860 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:30.594570 19860 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:30.594733 19860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:30.597368 19860 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8f0e0ff6a04870a8ad685c4213a90e" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35421 } }
I20260924 15:59:30.597903 19860 leader_election.cc:302] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb8f0e0ff6a04870a8ad685c4213a90e; no voters:
I20260924 15:59:30.598634 19860 leader_election.cc:288] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:30.598798 19865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:30.599495 19865 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [term 1 LEADER]: Becoming Leader. State: Replica: eb8f0e0ff6a04870a8ad685c4213a90e, State: Running, Role: LEADER
I20260924 15:59:30.600085 19865 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8f0e0ff6a04870a8ad685c4213a90e" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35421 } }
I20260924 15:59:30.600950 19860 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:30.603053 19867 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8f0e0ff6a04870a8ad685c4213a90e. Latest consensus state: current_term: 1 leader_uuid: "eb8f0e0ff6a04870a8ad685c4213a90e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8f0e0ff6a04870a8ad685c4213a90e" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35421 } } }
I20260924 15:59:30.603303 19866 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb8f0e0ff6a04870a8ad685c4213a90e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8f0e0ff6a04870a8ad685c4213a90e" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35421 } } }
I20260924 15:59:30.603564 19867 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:30.603629 19866 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:30.605464 19873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:30.611686 19873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:30.620988 19873 catalog_manager.cc:1347] Generated new cluster ID: 7362202cead642599602a74bbf357404
I20260924 15:59:30.621115 19873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:30.640941 19873 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:30.642913 19873 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:30.660876 19873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb8f0e0ff6a04870a8ad685c4213a90e: Generated new TSK 0
I20260924 15:59:30.661811 19873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 15:59:30.720208 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35421
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:31.056622 19886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:31.056998 19886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:31.057168 19886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:31.064935 19886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:31.065104 19886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:31.065300 19886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:31.075634 19886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:31.077082 19886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:31.078559 19886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:31.092916 19892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:31.093003 19894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:31.092948 19891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:31.093884 19886 server_base.cc:1034] running on GCE node
I20260924 15:59:31.094638 19886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:31.095660 19886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:31.096903 19886 hybrid_clock.cc:648] HybridClock initialized: now 1790265571096860 us; error 44 us; skew 500 ppm
I20260924 15:59:31.097235 19886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:31 dist-test-slave-xn4h krb5kdc[19779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:31.100426 19886 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 15:59:31.102020 19886 webserver.cc:460] Webserver started at http://127.16.233.129:44951/ using document root <none> and password file <none>
I20260924 15:59:31.102846 19886 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:31.102953 19886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:31.103209 19886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:31.106228 19886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "14023202c6c74d8a85e23598fdf0d4f9"
format_stamp: "Formatted at 2026-09-24 15:59:31 on dist-test-slave-xn4h"
server_key: "a3d7945da3ede7ced913d6e2c270c3e7"
server_key_iv: "c41b9b68d6f9563797a861553786940b"
server_key_version: "encryptionkey@0"
I20260924 15:59:31.107156 19886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "14023202c6c74d8a85e23598fdf0d4f9"
format_stamp: "Formatted at 2026-09-24 15:59:31 on dist-test-slave-xn4h"
server_key: "a3d7945da3ede7ced913d6e2c270c3e7"
server_key_iv: "c41b9b68d6f9563797a861553786940b"
server_key_version: "encryptionkey@0"
I20260924 15:59:31.112746 19886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 15:59:31.116412 19901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:31.117802 19886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 15:59:31.117981 19886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "14023202c6c74d8a85e23598fdf0d4f9"
format_stamp: "Formatted at 2026-09-24 15:59:31 on dist-test-slave-xn4h"
server_key: "a3d7945da3ede7ced913d6e2c270c3e7"
server_key_iv: "c41b9b68d6f9563797a861553786940b"
server_key_version: "encryptionkey@0"
I20260924 15:59:31.118198 19886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:31.136721 19886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:31.141218 19886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:31.141534 19886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:31.142635 19886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:31.144021 19886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:31.144134 19886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:31.144273 19886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:31.144359 19886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:31.177600 19886 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:44411
I20260924 15:59:31.177647 20014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:44411 every 8 connection(s)
I20260924 15:59:31.179000 19886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:31.181082 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 19886
I20260924 15:59:31.181300 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:31.182051 17318 external_mini_cluster.cc:1428] Setting key 89fdbe7789c7cde4f339fcc8e85ae9cd
Sep 24 15:59:31 dist-test-slave-xn4h krb5kdc[19779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 15:59:31.201879 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:31.181809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:44301 (local address 127.16.233.190:35421)
0924 15:59:31.182358 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:31.182365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:31.183452 (+ 1087us) server_negotiation.cc:407] Connection header received
0924 15:59:31.185410 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:31.185415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:31.185504 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:31.185628 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:31.187289 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:31.188038 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:31.190553 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:31.190978 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:31.191097 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:31.195586 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:31.195591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:31.195595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:31.195631 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:31.198445 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:31.199257 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:31.199262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:31.199266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:31.199399 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:31.199859 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:31.199863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:31.199866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:31.200140 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:31.200331 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:31.201201 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 15:59:31.201449 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":309,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
I20260924 15:59:31.203087 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:31.182461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:35421 (local address 127.16.233.129:44301)
0924 15:59:31.183302 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:31.183342 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:31.185133 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:31.185892 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:31.185901 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:31.186521 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:31.187117 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:31.187135 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:31.188211 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:31.188215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:31.190390 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:31.190410 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:31.191239 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:31.191249 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:31.191599 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:31.192409 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:31.192450 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:31.195303 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:31.198607 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:31.198619 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:31.198639 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:31.199106 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:31.199517 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:31.199522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:31.199526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:31.199732 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:31.200290 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:31.200301 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:31.200729 (+ 428us) client_negotiation.cc:769] Sending connection context
0924 15:59:31.201048 (+ 319us) client_negotiation.cc:240] Negotiation successful
0924 15:59:31.201689 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":323,"threads_started":1}
I20260924 15:59:31.204723 20015 heartbeater.cc:347] Connected to a master server at 127.16.233.190:35421
I20260924 15:59:31.205075 20015 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:31.206015 20015 heartbeater.cc:511] Master 127.16.233.190:35421 requested a full tablet report, sending...
I20260924 15:59:31.208567 19825 ts_manager.cc:194] Registered new tserver with Master: 14023202c6c74d8a85e23598fdf0d4f9 (127.16.233.129:44411)
I20260924 15:59:31.210222 19825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:44301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 15:59:31.227633 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35421
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:31.554380 20021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:31.554768 20021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:31.554924 20021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:31.567026 20021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:31.567180 20021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:31.567369 20021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:31.582031 20021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:31.583422 20021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:31.584753 20021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:31.598973 20026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:31.598908 20029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:31.598950 20027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:31.599783 20021 server_base.cc:1034] running on GCE node
I20260924 15:59:31.600414 20021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:31.601449 20021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:31.602645 20021 hybrid_clock.cc:648] HybridClock initialized: now 1790265571602577 us; error 57 us; skew 500 ppm
I20260924 15:59:31.602952 20021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:31 dist-test-slave-xn4h krb5kdc[19779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:31.614675 20021 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 15:59:31.616587 20021 webserver.cc:460] Webserver started at http://127.16.233.130:42613/ using document root <none> and password file <none>
I20260924 15:59:31.617321 20021 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:31.617430 20021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:31.617700 20021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:31.620873 20021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "00c169894ced44cea0abe0693532b402"
format_stamp: "Formatted at 2026-09-24 15:59:31 on dist-test-slave-xn4h"
server_key: "f0594e0183728d7ea99f5a77b12fc825"
server_key_iv: "3244a84e1c51a6efbaee88db649aea37"
server_key_version: "encryptionkey@0"
I20260924 15:59:31.621744 20021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "00c169894ced44cea0abe0693532b402"
format_stamp: "Formatted at 2026-09-24 15:59:31 on dist-test-slave-xn4h"
server_key: "f0594e0183728d7ea99f5a77b12fc825"
server_key_iv: "3244a84e1c51a6efbaee88db649aea37"
server_key_version: "encryptionkey@0"
I20260924 15:59:31.627435 20021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 15:59:31.631218 20036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:31.632622 20021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:31.632788 20021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "00c169894ced44cea0abe0693532b402"
format_stamp: "Formatted at 2026-09-24 15:59:31 on dist-test-slave-xn4h"
server_key: "f0594e0183728d7ea99f5a77b12fc825"
server_key_iv: "3244a84e1c51a6efbaee88db649aea37"
server_key_version: "encryptionkey@0"
I20260924 15:59:31.632972 20021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:31.647445 20021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:31.651711 20021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:31.651986 20021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:31.652940 20021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:31.654321 20021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:31.654467 20021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:31.654575 20021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:31.654625 20021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:31.688984 20021 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:34425
I20260924 15:59:31.689059 20149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:34425 every 8 connection(s)
I20260924 15:59:31.690287 20021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:31.696834 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 20021
I20260924 15:59:31.697038 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:31.697484 17318 external_mini_cluster.cc:1428] Setting key da73642ba958a75483b5705d9b05e20f
Sep 24 15:59:31 dist-test-slave-xn4h krb5kdc[19779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:31.712047 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:31.692891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:44877 (local address 127.16.233.190:35421)
0924 15:59:31.693068 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:31.693072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:31.694271 (+ 1199us) server_negotiation.cc:407] Connection header received
0924 15:59:31.696142 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:31.696147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:31.696243 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:31.696351 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:31.698065 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:31.698925 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:31.701192 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:31.701706 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:31.701843 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:31.706275 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:31.706280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:31.706283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:31.706364 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:31.708907 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:31.709652 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:31.709662 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:31.709664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:31.709795 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:31.710232 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:31.710238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:31.710241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:31.710570 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:31.710723 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:31.711438 (+ 715us) server_negotiation.cc:299] Negotiation successful
0924 15:59:31.711678 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 15:59:31.713050 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:31.693422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:35421 (local address 127.16.233.130:44877)
0924 15:59:31.694107 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:31.694149 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:31.695863 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:31.696616 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:31.696625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:31.697245 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:31.697892 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:31.697911 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:31.699091 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:31.699096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:31.701033 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:31.701052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:31.701968 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:31.701977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:31.702340 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:31.703117 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:31.703154 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:31.706038 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:31.709056 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:31.709065 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:31.709088 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:31.709523 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:31.709914 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:31.709919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:31.709923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:31.710121 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:31.710695 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:31.710704 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:31.711069 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 15:59:31.711345 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 15:59:31.711914 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":268,"threads_started":1}
I20260924 15:59:31.714679 20150 heartbeater.cc:347] Connected to a master server at 127.16.233.190:35421
I20260924 15:59:31.715070 20150 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:31.716118 20150 heartbeater.cc:511] Master 127.16.233.190:35421 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 15:59:31.717921 19825 ts_manager.cc:194] Registered new tserver with Master: 00c169894ced44cea0abe0693532b402 (127.16.233.130:34425)
I20260924 15:59:31.718899 19825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:44877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 15:59:31.741634 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35421
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:32.075531 20155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:32.075850 20155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:32.076025 20155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:32.083637 20155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:32.083765 20155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:32.083945 20155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:32.094074 20155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:32.095574 20155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:32.097014 20155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:32.110631 20163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:32.110596 20161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:32.110800 20160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:32.111490 20155 server_base.cc:1034] running on GCE node
I20260924 15:59:32.112154 20155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:32.113099 20155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:32.114367 20155 hybrid_clock.cc:648] HybridClock initialized: now 1790265572114274 us; error 41 us; skew 500 ppm
I20260924 15:59:32.114739 20155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:32 dist-test-slave-xn4h krb5kdc[19779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265572, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:32.118732 20155 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 15:59:32.122134 20155 webserver.cc:460] Webserver started at http://127.16.233.131:42985/ using document root <none> and password file <none>
I20260924 15:59:32.123082 20155 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:32.123229 20155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:32.123602 20155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:32.127504 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "d3be32c54afc4b80842bfb21fb09a3a7"
format_stamp: "Formatted at 2026-09-24 15:59:32 on dist-test-slave-xn4h"
server_key: "3b4b30b613e3ef9565e3cb548083461b"
server_key_iv: "8ac797962b812f50efdf9ea0852f3b91"
server_key_version: "encryptionkey@0"
I20260924 15:59:32.128360 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "d3be32c54afc4b80842bfb21fb09a3a7"
format_stamp: "Formatted at 2026-09-24 15:59:32 on dist-test-slave-xn4h"
server_key: "3b4b30b613e3ef9565e3cb548083461b"
server_key_iv: "8ac797962b812f50efdf9ea0852f3b91"
server_key_version: "encryptionkey@0"
I20260924 15:59:32.133846 20155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 15:59:32.137733 20170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:32.139295 20155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:32.139474 20155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "d3be32c54afc4b80842bfb21fb09a3a7"
format_stamp: "Formatted at 2026-09-24 15:59:32 on dist-test-slave-xn4h"
server_key: "3b4b30b613e3ef9565e3cb548083461b"
server_key_iv: "8ac797962b812f50efdf9ea0852f3b91"
server_key_version: "encryptionkey@0"
I20260924 15:59:32.139691 20155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:32.172940 20155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:32.177546 20155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:32.177894 20155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:32.179157 20155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:32.180913 20155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:32.181010 20155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:32.181134 20155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:32.181198 20155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:32.213276 20015 heartbeater.cc:503] Master 127.16.233.190:35421 was elected leader, sending a full tablet report...
I20260924 15:59:32.215610 20155 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:40973
I20260924 15:59:32.215646 20283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:40973 every 8 connection(s)
I20260924 15:59:32.217029 20155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:32.220489 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 20155
I20260924 15:59:32.220710 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:32.221287 17318 external_mini_cluster.cc:1428] Setting key 11611a9c39c9c5bf4fc9e17eaaa96c31
Sep 24 15:59:32 dist-test-slave-xn4h krb5kdc[19779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265572, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:32.238448 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:32.219561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:44217 (local address 127.16.233.190:35421)
0924 15:59:32.220034 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:32.220038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:32.221900 (+ 1862us) server_negotiation.cc:407] Connection header received
0924 15:59:32.223727 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:32.223732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:32.223819 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:32.223940 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:32.225568 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:32.226448 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:32.228046 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:32.228467 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:32.228600 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:32.232235 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:32.232240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:32.232244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:32.232279 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:32.234922 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:32.235894 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:32.235898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:32.235901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:32.236024 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:32.236569 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:32.236572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:32.236574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:32.237013 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:32.237134 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:32.237921 (+ 787us) server_negotiation.cc:299] Negotiation successful
0924 15:59:32.238095 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":271,"threads_started":1}
I20260924 15:59:32.239553 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:32.220100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:35421 (local address 127.16.233.131:44217)
0924 15:59:32.220971 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:32.221031 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:32.223444 (+ 2413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:32.224222 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:32.224234 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:32.224831 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:32.225400 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:32.225418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:32.226606 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:32.226612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:32.227897 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:32.227910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:32.228722 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:32.228730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:32.228972 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:32.229626 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:32.229653 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:32.232011 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:32.235070 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:32.235078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:32.235092 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:32.235770 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:32.236167 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:32.236172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:32.236176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:32.236388 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:32.237150 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:32.237159 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:32.237554 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 15:59:32.237809 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 15:59:32.238212 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":313,"threads_started":1}
I20260924 15:59:32.240921 20284 heartbeater.cc:347] Connected to a master server at 127.16.233.190:35421
I20260924 15:59:32.241236 20284 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:32.241932 20284 heartbeater.cc:511] Master 127.16.233.190:35421 requested a full tablet report, sending...
I20260924 15:59:32.243362 19825 ts_manager.cc:194] Registered new tserver with Master: d3be32c54afc4b80842bfb21fb09a3a7 (127.16.233.131:40973)
I20260924 15:59:32.244160 19825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:44217
I20260924 15:59:32.249198 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:59:32.282043 20286 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:59:32.279383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.16.233.190:35421)
0924 15:59:32.279635 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:32.279645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:32.279991 (+ 346us) server_negotiation.cc:407] Connection header received
0924 15:59:32.280271 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:32.280275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:32.280361 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:32.280469 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:32.281766 (+ 1297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51680: BlockingRecv error: recv got EOF from 127.0.0.1:51680 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 15:59:32.285012 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19886
W20260924 15:59:32.288043 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19886: Not authorized: Client connection negotiation failed: client connection to 127.16.233.129:44411: server requires authentication, but client does not have Kerberos credentials available
I20260924 15:59:32.288290 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 19886
I20260924 15:59:32.304704 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20021
W20260924 15:59:32.307416 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20021: Not authorized: Client connection negotiation failed: client connection to 127.16.233.130:34425: server requires authentication, but client does not have Kerberos credentials available
I20260924 15:59:32.307606 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 20021
I20260924 15:59:32.324247 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20155
W20260924 15:59:32.327220 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20155: Not authorized: Client connection negotiation failed: client connection to 127.16.233.131:40973: server requires authentication, but client does not have Kerberos credentials available
I20260924 15:59:32.327395 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 20155
I20260924 15:59:32.343500 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 19792
I20260924 15:59:32.344794 19855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:32.449287 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 19792
2026-09-24T15:59:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3628 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:32 dist-test-slave-xn4h krb5kdc[20302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:32 dist-test-slave-xn4h krb5kdc[20302](info): set up 2 sockets
Sep 24 15:59:32 dist-test-slave-xn4h krb5kdc[20302](info): commencing operation
krb5kdc: starting...
W20260924 15:59:33.544319 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:33 dist-test-slave-xn4h krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:33.746034 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:43283
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:43283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:34.088034 20315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:34.088346 20315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:34.088438 20315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:34.095952 20315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:34.096071 20315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:34.096117 20315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:34.096166 20315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:34.096216 20315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:34.106146 20315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:43283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:34.107496 20315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:34.108783 20315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:34.126534 20321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:34.126861 20323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:34.127429 20320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:34.127928 20315 server_base.cc:1034] running on GCE node
I20260924 15:59:34.129036 20315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:34.130834 20315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:34.132050 20315 hybrid_clock.cc:648] HybridClock initialized: now 1790265574132009 us; error 34 us; skew 500 ppm
I20260924 15:59:34.132402 20315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:34 dist-test-slave-xn4h krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265574, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:34.135535 20315 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:34.137005 20315 webserver.cc:460] Webserver started at http://127.16.233.190:37295/ using document root <none> and password file <none>
I20260924 15:59:34.137672 20315 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:34.137774 20315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:34.138036 20315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:34.141108 20315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "7b33732d453c4a4cb11a2534384a77f5"
format_stamp: "Formatted at 2026-09-24 15:59:34 on dist-test-slave-xn4h"
server_key: "a201e24f229b6f2d399a11d05f59938d"
server_key_iv: "56191db72122c1d3e6d96e1c5c04dbfa"
server_key_version: "encryptionkey@0"
I20260924 15:59:34.141857 20315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "7b33732d453c4a4cb11a2534384a77f5"
format_stamp: "Formatted at 2026-09-24 15:59:34 on dist-test-slave-xn4h"
server_key: "a201e24f229b6f2d399a11d05f59938d"
server_key_iv: "56191db72122c1d3e6d96e1c5c04dbfa"
server_key_version: "encryptionkey@0"
I20260924 15:59:34.147326 20315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 15:59:34.150789 20330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:34.152185 20315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:34.152367 20315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "7b33732d453c4a4cb11a2534384a77f5"
format_stamp: "Formatted at 2026-09-24 15:59:34 on dist-test-slave-xn4h"
server_key: "a201e24f229b6f2d399a11d05f59938d"
server_key_iv: "56191db72122c1d3e6d96e1c5c04dbfa"
server_key_version: "encryptionkey@0"
I20260924 15:59:34.152573 20315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:34.173866 20315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:34.178354 20315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:34.178663 20315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:34.197203 20315 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:43283
I20260924 15:59:34.197235 20381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:43283 every 8 connection(s)
I20260924 15:59:34.198637 20315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:34.202258 20382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:34.206018 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 20315
I20260924 15:59:34.206250 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:34.207080 17318 external_mini_cluster.cc:1428] Setting key 882bc86508b1450713b03bfa7573b9a7
I20260924 15:59:34.215760 20382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5: Bootstrap starting.
I20260924 15:59:34.219683 20382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:34.221087 20382 log.cc:824] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5: Log is configured to *not* fsync() on all Append() calls
Sep 24 15:59:34 dist-test-slave-xn4h krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:34.224228 20382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5: No bootstrap required, opened a new log
I20260924 15:59:34.230005 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:34.208979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.16.233.190:43283)
0924 15:59:34.209743 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:34.209754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:34.209795 (+ 41us) server_negotiation.cc:407] Connection header received
0924 15:59:34.211463 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:34.211486 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:34.211907 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:34.212247 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:34.214097 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:34.215465 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:34.217202 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:34.217916 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:34.218046 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:34.222869 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:34.222877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:34.222892 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:34.222931 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:34.225932 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:34.226806 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:34.226812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:34.226818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:34.226989 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:34.227359 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:34.227362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:34.227364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:34.227902 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:34.228121 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:34.228562 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 15:59:34.228960 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260924 15:59:34.233554 20382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b33732d453c4a4cb11a2534384a77f5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43283 } }
I20260924 15:59:34.233887 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:59:34.233987 20382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:34.234088 20382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b33732d453c4a4cb11a2534384a77f5, State: Initialized, Role: FOLLOWER
I20260924 15:59:34.234679 20382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b33732d453c4a4cb11a2534384a77f5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43283 } }
I20260924 15:59:34.234917 20382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:34.235057 20382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:34.235220 20382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:34.237380 20382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b33732d453c4a4cb11a2534384a77f5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43283 } }
I20260924 15:59:34.237885 20382 leader_election.cc:302] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b33732d453c4a4cb11a2534384a77f5; no voters:
I20260924 15:59:34.238493 20382 leader_election.cc:288] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:34.238637 20387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:34.239846 20387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [term 1 LEADER]: Becoming Leader. State: Replica: 7b33732d453c4a4cb11a2534384a77f5, State: Running, Role: LEADER
I20260924 15:59:34.240466 20387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b33732d453c4a4cb11a2534384a77f5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43283 } }
I20260924 15:59:34.240828 20382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:34.244453 20388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b33732d453c4a4cb11a2534384a77f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b33732d453c4a4cb11a2534384a77f5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43283 } } }
I20260924 15:59:34.244855 20388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:34.244009 20389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b33732d453c4a4cb11a2534384a77f5. Latest consensus state: current_term: 1 leader_uuid: "7b33732d453c4a4cb11a2534384a77f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b33732d453c4a4cb11a2534384a77f5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43283 } } }
I20260924 15:59:34.245417 20389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:34.262614 20394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:34.267970 20394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:34.278885 20394 catalog_manager.cc:1347] Generated new cluster ID: 238c51f38f324acd95ef04cf83d993f4
I20260924 15:59:34.279006 20394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:34.284068 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:34.275627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.16.233.190:43283)
0924 15:59:34.275866 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:34.275875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:34.276499 (+ 624us) server_negotiation.cc:407] Connection header received
0924 15:59:34.276786 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:34.276792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:34.276901 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:34.277014 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:34.278340 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:34.279423 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:34.280763 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:34.281199 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:34.281340 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:34.282473 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:34.282478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:34.282481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:34.283213 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:34.283284 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:34.283313 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:34.283427 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:34.283569 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 15:59:34.283676 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 15:59:34.300859 20394 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:34.302224 20394 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:34.310937 20394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b33732d453c4a4cb11a2534384a77f5: Generated new TSK 0
I20260924 15:59:34.311878 20394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:59:34.329594 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20315
I20260924 15:59:34.331270 20377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:34.425390 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 20315
2026-09-24T15:59:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1970 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:34 dist-test-slave-xn4h krb5kdc[20416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:34 dist-test-slave-xn4h krb5kdc[20416](info): set up 2 sockets
Sep 24 15:59:34 dist-test-slave-xn4h krb5kdc[20416](info): commencing operation
krb5kdc: starting...
W20260924 15:59:35.520027 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:35 dist-test-slave-xn4h krb5kdc[20416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:35.718601 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45167
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:36.049937 20429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:36.050258 20429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:36.050442 20429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:36.057859 20429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:36.057976 20429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:36.058068 20429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:36.058147 20429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:36.058223 20429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:36.068483 20429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:45167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:36.069918 20429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:36.071317 20429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:36.083297 20435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:36.083387 20434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:36.083298 20437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:36.084224 20429 server_base.cc:1034] running on GCE node
I20260924 15:59:36.085040 20429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:36.086638 20429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:36.087857 20429 hybrid_clock.cc:648] HybridClock initialized: now 1790265576087812 us; error 39 us; skew 500 ppm
I20260924 15:59:36.088222 20429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:36 dist-test-slave-xn4h krb5kdc[20416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265576, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:36.091367 20429 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:36.092960 20429 webserver.cc:460] Webserver started at http://127.16.233.190:45615/ using document root <none> and password file <none>
I20260924 15:59:36.093679 20429 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:36.093822 20429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:36.094125 20429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:36.097210 20429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "792df4d7b7c449c294d67e1bcd52b96f"
format_stamp: "Formatted at 2026-09-24 15:59:36 on dist-test-slave-xn4h"
server_key: "0aae9167230ebcca7721b88fc8eb231c"
server_key_iv: "02b0a4624c95dc58d01915cdac1337c6"
server_key_version: "encryptionkey@0"
I20260924 15:59:36.097990 20429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "792df4d7b7c449c294d67e1bcd52b96f"
format_stamp: "Formatted at 2026-09-24 15:59:36 on dist-test-slave-xn4h"
server_key: "0aae9167230ebcca7721b88fc8eb231c"
server_key_iv: "02b0a4624c95dc58d01915cdac1337c6"
server_key_version: "encryptionkey@0"
I20260924 15:59:36.103714 20429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 15:59:36.107331 20444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:36.108780 20429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 15:59:36.108973 20429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "792df4d7b7c449c294d67e1bcd52b96f"
format_stamp: "Formatted at 2026-09-24 15:59:36 on dist-test-slave-xn4h"
server_key: "0aae9167230ebcca7721b88fc8eb231c"
server_key_iv: "02b0a4624c95dc58d01915cdac1337c6"
server_key_version: "encryptionkey@0"
I20260924 15:59:36.109186 20429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:36.128764 20429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:36.133234 20429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:36.133551 20429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:36.153272 20429 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:45167
I20260924 15:59:36.153322 20495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:45167 every 8 connection(s)
I20260924 15:59:36.154817 20429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:36.158210 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 20429
I20260924 15:59:36.158461 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:36.159233 20496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:36.159304 17318 external_mini_cluster.cc:1428] Setting key 2084bb4d092496e05d0b92a5e2c10936
Sep 24 15:59:36 dist-test-slave-xn4h krb5kdc[20416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:36.174533 20496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f: Bootstrap starting.
I20260924 15:59:36.178395 20496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:36.179636 20496 log.cc:824] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:36.183208 20496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f: No bootstrap required, opened a new log
I20260924 15:59:36.183884 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:36.161392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.16.233.190:45167)
0924 15:59:36.162276 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:36.162287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:36.162388 (+ 101us) server_negotiation.cc:407] Connection header received
0924 15:59:36.164217 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:36.164241 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:36.164666 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:36.165037 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:36.166807 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:36.168112 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:36.169920 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:36.170720 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:36.170859 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:36.175856 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:36.175864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:36.175880 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:36.175921 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:36.179136 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:36.180018 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:36.180024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:36.180031 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:36.180273 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:36.180706 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:36.180714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:36.180720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:36.181461 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:36.181699 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:36.182250 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 15:59:36.182786 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":403,"threads_started":1}
I20260924 15:59:36.190847 20496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792df4d7b7c449c294d67e1bcd52b96f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45167 } }
I20260924 15:59:36.191258 20496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:36.191398 20496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792df4d7b7c449c294d67e1bcd52b96f, State: Initialized, Role: FOLLOWER
I20260924 15:59:36.191951 20496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792df4d7b7c449c294d67e1bcd52b96f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45167 } }
I20260924 15:59:36.192193 20496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:36.192344 20496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:36.192525 20496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:36.194821 20496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792df4d7b7c449c294d67e1bcd52b96f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45167 } }
I20260924 15:59:36.195340 20496 leader_election.cc:302] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 792df4d7b7c449c294d67e1bcd52b96f; no voters:
I20260924 15:59:36.195941 20496 leader_election.cc:288] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:36.196081 20502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:36.196780 20502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [term 1 LEADER]: Becoming Leader. State: Replica: 792df4d7b7c449c294d67e1bcd52b96f, State: Running, Role: LEADER
I20260924 15:59:36.197377 20502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792df4d7b7c449c294d67e1bcd52b96f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45167 } }
I20260924 15:59:36.198081 20496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:36.200039 20503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "792df4d7b7c449c294d67e1bcd52b96f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792df4d7b7c449c294d67e1bcd52b96f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45167 } } }
I20260924 15:59:36.200420 20503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:36.201306 20507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:36.201157 20504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 792df4d7b7c449c294d67e1bcd52b96f. Latest consensus state: current_term: 1 leader_uuid: "792df4d7b7c449c294d67e1bcd52b96f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792df4d7b7c449c294d67e1bcd52b96f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45167 } } }
I20260924 15:59:36.202468 20504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:36.207286 20507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:36.216326 20507 catalog_manager.cc:1347] Generated new cluster ID: bc97e4456eff4bfe9f4ee59f318487a2
I20260924 15:59:36.216459 20507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:36.238658 20507 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:36.240038 20507 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:36.245632 20507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 792df4d7b7c449c294d67e1bcd52b96f: Generated new TSK 0
I20260924 15:59:36.246631 20507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 15:59:36.305650 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45167
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:36.645977 20523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:36.646271 20523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:36.646435 20523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:36.653971 20523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:36.654095 20523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:36.654208 20523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:36.664284 20523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:36.665633 20523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:36.667019 20523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:36.680380 20532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:36.680403 20530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:36.680467 20528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:36.681126 20523 server_base.cc:1034] running on GCE node
I20260924 15:59:36.681773 20523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:36.682883 20523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:36.684108 20523 hybrid_clock.cc:648] HybridClock initialized: now 1790265576684041 us; error 60 us; skew 500 ppm
I20260924 15:59:36.684468 20523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:36 dist-test-slave-xn4h krb5kdc[20416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265576, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:36.694058 20523 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 15:59:36.695802 20523 webserver.cc:460] Webserver started at http://127.16.233.129:45411/ using document root <none> and password file <none>
I20260924 15:59:36.696642 20523 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:36.696802 20523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:36.697126 20523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:36.700403 20523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "ecae1e9966f1440b9e980c030333c810"
format_stamp: "Formatted at 2026-09-24 15:59:36 on dist-test-slave-xn4h"
server_key: "3ddcaa3e5db09305c13464eda7a58096"
server_key_iv: "c99ea18cefb19a5a72a9425d759241bf"
server_key_version: "encryptionkey@0"
I20260924 15:59:36.701237 20523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "ecae1e9966f1440b9e980c030333c810"
format_stamp: "Formatted at 2026-09-24 15:59:36 on dist-test-slave-xn4h"
server_key: "3ddcaa3e5db09305c13464eda7a58096"
server_key_iv: "c99ea18cefb19a5a72a9425d759241bf"
server_key_version: "encryptionkey@0"
I20260924 15:59:36.706866 20523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 15:59:36.710382 20539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:36.711745 20523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:36.711933 20523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "ecae1e9966f1440b9e980c030333c810"
format_stamp: "Formatted at 2026-09-24 15:59:36 on dist-test-slave-xn4h"
server_key: "3ddcaa3e5db09305c13464eda7a58096"
server_key_iv: "c99ea18cefb19a5a72a9425d759241bf"
server_key_version: "encryptionkey@0"
I20260924 15:59:36.712153 20523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:36.733539 20523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:36.737776 20523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:36.738089 20523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:36.739130 20523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:36.740510 20523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:36.740615 20523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:36.740767 20523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:36.740840 20523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:36.774029 20523 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:45447
I20260924 15:59:36.774076 20652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:45447 every 8 connection(s)
I20260924 15:59:36.775343 20523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:36.777902 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 20523
I20260924 15:59:36.778131 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:36.778882 17318 external_mini_cluster.cc:1428] Setting key 17f68014779ab92feb1e4ec78d8faabc
Sep 24 15:59:36 dist-test-slave-xn4h krb5kdc[20416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265576, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:36.796532 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:36.777923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:36923 (local address 127.16.233.190:45167)
0924 15:59:36.778467 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:36.778475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:36.779481 (+ 1006us) server_negotiation.cc:407] Connection header received
0924 15:59:36.781265 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:36.781272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:36.781357 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:36.781481 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:36.782966 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:36.784130 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:36.785649 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:36.786058 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:36.786156 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:36.790660 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:36.790664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:36.790668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:36.790700 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:36.793137 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:36.794113 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:36.794117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:36.794119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:36.794251 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:36.794793 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:36.794796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:36.794798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:36.795049 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:36.795156 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:36.796104 (+ 948us) server_negotiation.cc:299] Negotiation successful
0924 15:59:36.796283 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":241,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 15:59:36.798094 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:36.778480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45167 (local address 127.16.233.129:36923)
0924 15:59:36.779291 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:36.779336 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:36.781001 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:36.781669 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:36.781677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:36.782172 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:36.782793 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:36.782817 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:36.784294 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:36.784297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:36.785495 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:36.785507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:36.786298 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:36.786330 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:36.786660 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:36.787403 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:36.787442 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:36.790444 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:36.793298 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:36.793310 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:36.793328 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:36.793988 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:36.794396 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:36.794404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:36.794409 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:36.794674 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:36.795183 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:36.795194 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:36.795672 (+ 478us) client_negotiation.cc:769] Sending connection context
0924 15:59:36.796020 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 15:59:36.796608 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":346,"threads_started":1}
I20260924 15:59:36.799746 20653 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45167
I20260924 15:59:36.800060 20653 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
I20260924 15:59:36.801019 20653 heartbeater.cc:511] Master 127.16.233.190:45167 requested a full tablet report, sending...
I20260924 15:59:36.803493 20461 ts_manager.cc:194] Registered new tserver with Master: ecae1e9966f1440b9e980c030333c810 (127.16.233.129:45447)
I20260924 15:59:36.805683 20461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:36923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 15:59:36.826185 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45167
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:37.149398 20659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:37.149751 20659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:37.149930 20659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:37.160243 20659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:37.160382 20659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:37.160584 20659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:37.175513 20659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:37.176847 20659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:37.178153 20659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:37.191568 20664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:37.191502 20665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:37.191527 20667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:37.192917 20659 server_base.cc:1034] running on GCE node
I20260924 15:59:37.193492 20659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:37.194465 20659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:37.195677 20659 hybrid_clock.cc:648] HybridClock initialized: now 1790265577195608 us; error 61 us; skew 500 ppm
I20260924 15:59:37.196007 20659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:37 dist-test-slave-xn4h krb5kdc[20416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:37.199806 20659 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 15:59:37.201292 20659 webserver.cc:460] Webserver started at http://127.16.233.130:34295/ using document root <none> and password file <none>
I20260924 15:59:37.202016 20659 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:37.202170 20659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:37.202544 20659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:37.205597 20659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "73838fdcdb4a438ab3c274bd319f4da8"
format_stamp: "Formatted at 2026-09-24 15:59:37 on dist-test-slave-xn4h"
server_key: "53bb2f9cc8dba093f17c38d4bd03770f"
server_key_iv: "86fef6741605f969ae4cfd6c335c8aac"
server_key_version: "encryptionkey@0"
I20260924 15:59:37.206488 20659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "73838fdcdb4a438ab3c274bd319f4da8"
format_stamp: "Formatted at 2026-09-24 15:59:37 on dist-test-slave-xn4h"
server_key: "53bb2f9cc8dba093f17c38d4bd03770f"
server_key_iv: "86fef6741605f969ae4cfd6c335c8aac"
server_key_version: "encryptionkey@0"
I20260924 15:59:37.211920 20659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 15:59:37.216073 20674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:37.217604 20659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 15:59:37.217790 20659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "73838fdcdb4a438ab3c274bd319f4da8"
format_stamp: "Formatted at 2026-09-24 15:59:37 on dist-test-slave-xn4h"
server_key: "53bb2f9cc8dba093f17c38d4bd03770f"
server_key_iv: "86fef6741605f969ae4cfd6c335c8aac"
server_key_version: "encryptionkey@0"
I20260924 15:59:37.218008 20659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:37.234247 20659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:37.238814 20659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:37.239133 20659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:37.240168 20659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:37.241570 20659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:37.241675 20659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:37.241830 20659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:37.241909 20659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:37.274915 20659 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:39463
I20260924 15:59:37.274935 20787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:39463 every 8 connection(s)
I20260924 15:59:37.276278 20659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:37.285178 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 20659
I20260924 15:59:37.285363 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:37.285820 17318 external_mini_cluster.cc:1428] Setting key 799105b6e2f18ab9db5612fe97295d25
Sep 24 15:59:37 dist-test-slave-xn4h krb5kdc[20416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:37.297108 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:37.278833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:57649 (local address 127.16.233.190:45167)
0924 15:59:37.279004 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:37.279009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:37.280249 (+ 1240us) server_negotiation.cc:407] Connection header received
0924 15:59:37.282032 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:37.282036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:37.282110 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:37.282229 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:37.283729 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.284420 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.286181 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.286838 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.286990 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:37.291098 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:37.291103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:37.291106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:37.291142 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:37.293819 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.294714 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.294724 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.294726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.294869 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.295337 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.295341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.295343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.295624 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:37.295786 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:37.296518 (+ 732us) server_negotiation.cc:299] Negotiation successful
0924 15:59:37.296753 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 15:59:37.298148 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:37.279211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45167 (local address 127.16.233.130:57649)
0924 15:59:37.280068 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:37.280112 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:37.281784 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:37.282484 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:37.282492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:37.282997 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:37.283571 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:37.283589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.284558 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.284561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:37.285996 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:37.286010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.287064 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.287069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:37.287285 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:37.288075 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:37.288105 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:37.290831 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:37.293996 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:37.294005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:37.294024 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:37.294572 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:37.294989 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:37.294994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:37.294998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:37.295196 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:37.295758 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:37.295766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:37.296144 (+ 378us) client_negotiation.cc:769] Sending connection context
0924 15:59:37.296416 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 15:59:37.297002 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":354,"threads_started":1}
I20260924 15:59:37.299914 20788 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45167
I20260924 15:59:37.300242 20788 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:37.301010 20788 heartbeater.cc:511] Master 127.16.233.190:45167 requested a full tablet report, sending...
I20260924 15:59:37.302600 20461 ts_manager.cc:194] Registered new tserver with Master: 73838fdcdb4a438ab3c274bd319f4da8 (127.16.233.130:39463)
I20260924 15:59:37.303406 20461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:57649
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 15:59:37.330427 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45167
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:37.652125 20793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:37.652410 20793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:37.652607 20793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:37.659799 20793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:37.659935 20793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:37.660111 20793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:37.669904 20793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:37.671293 20793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:37.672788 20793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:37.686187 20798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:37.686213 20801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:37.686146 20799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:37.686959 20793 server_base.cc:1034] running on GCE node
I20260924 15:59:37.687532 20793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:37.688480 20793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:37.689709 20793 hybrid_clock.cc:648] HybridClock initialized: now 1790265577689663 us; error 58 us; skew 500 ppm
I20260924 15:59:37.690026 20793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:37 dist-test-slave-xn4h krb5kdc[20416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:37.693090 20793 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 15:59:37.694600 20793 webserver.cc:460] Webserver started at http://127.16.233.131:38719/ using document root <none> and password file <none>
I20260924 15:59:37.695276 20793 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:37.695420 20793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:37.695727 20793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:37.698832 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "9d0381ae15734a56aedc76d411211f12"
format_stamp: "Formatted at 2026-09-24 15:59:37 on dist-test-slave-xn4h"
server_key: "429c92fe5fad9d8585195c9cbd483fce"
server_key_iv: "38fa62dd07dd955234001557d7bac2d9"
server_key_version: "encryptionkey@0"
I20260924 15:59:37.699733 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "9d0381ae15734a56aedc76d411211f12"
format_stamp: "Formatted at 2026-09-24 15:59:37 on dist-test-slave-xn4h"
server_key: "429c92fe5fad9d8585195c9cbd483fce"
server_key_iv: "38fa62dd07dd955234001557d7bac2d9"
server_key_version: "encryptionkey@0"
I20260924 15:59:37.705214 20793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 15:59:37.708842 20808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:37.710150 20793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 15:59:37.710381 20793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "9d0381ae15734a56aedc76d411211f12"
format_stamp: "Formatted at 2026-09-24 15:59:37 on dist-test-slave-xn4h"
server_key: "429c92fe5fad9d8585195c9cbd483fce"
server_key_iv: "38fa62dd07dd955234001557d7bac2d9"
server_key_version: "encryptionkey@0"
I20260924 15:59:37.710589 20793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:37.724957 20793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:37.728915 20793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:37.729238 20793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:37.730204 20793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:37.731604 20793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:37.731717 20793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:37.731865 20793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:37.731941 20793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:37.765021 20793 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:39033
I20260924 15:59:37.765066 20921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:39033 every 8 connection(s)
I20260924 15:59:37.766405 20793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:37.769235 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 20793
I20260924 15:59:37.769418 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:37.770076 17318 external_mini_cluster.cc:1428] Setting key 68b6b8d47587b7afaf3376b6976215e4
Sep 24 15:59:37 dist-test-slave-xn4h krb5kdc[20416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:37.787969 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:37.768961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:49975 (local address 127.16.233.190:45167)
0924 15:59:37.769139 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:37.769145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:37.770681 (+ 1536us) server_negotiation.cc:407] Connection header received
0924 15:59:37.772725 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:37.772730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:37.772865 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:37.772979 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:37.775088 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.775833 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.778138 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.778621 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.778721 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:37.782130 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:37.782135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:37.782138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:37.782170 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:37.784980 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.785882 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.785886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.785888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.786012 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.786526 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.786530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.786532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.786807 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:37.786917 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:37.787565 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 15:59:37.787728 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 15:59:37.788914 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:37.769603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45167 (local address 127.16.233.131:49975)
0924 15:59:37.770505 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:37.770543 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:37.772469 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:37.773309 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:37.773322 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:37.774025 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:37.774894 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:37.774924 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.775985 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.775990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:37.778009 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:37.778024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.778858 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.778865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:37.779148 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:37.779819 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:37.779840 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:37.781931 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:37.785146 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:37.785156 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:37.785172 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:37.785738 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:37.786106 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:37.786111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:37.786131 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:37.786423 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:37.786914 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:37.786921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:37.787232 (+ 311us) client_negotiation.cc:769] Sending connection context
0924 15:59:37.787462 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 15:59:37.787851 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":335,"threads_started":1}
I20260924 15:59:37.790191 20922 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45167
I20260924 15:59:37.790534 20922 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:37.791229 20922 heartbeater.cc:511] Master 127.16.233.190:45167 requested a full tablet report, sending...
I20260924 15:59:37.792627 20461 ts_manager.cc:194] Registered new tserver with Master: 9d0381ae15734a56aedc76d411211f12 (127.16.233.131:39033)
I20260924 15:59:37.793435 20461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:49975
I20260924 15:59:37.798382 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:59:37.809484 20653 heartbeater.cc:503] Master 127.16.233.190:45167 was elected leader, sending a full tablet report...
Sep 24 15:59:37 dist-test-slave-xn4h krb5kdc[20416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265577, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 15:59:37 dist-test-slave-xn4h krb5kdc[20416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265577, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:37.847699 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:37.833597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.16.233.190:45167)
0924 15:59:37.833812 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:37.833821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:37.834255 (+ 434us) server_negotiation.cc:407] Connection header received
0924 15:59:37.834545 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:37.834549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:37.834616 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:37.834726 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:37.836154 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.836896 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.838159 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.838632 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.838774 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:37.842770 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:37.842775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:37.842778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:37.842818 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:37.845262 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.845942 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.845945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.845948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.846084 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.846476 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.846479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.846481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.846818 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:37.847009 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:37.847295 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 15:59:37.847476 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 15:59:37.848357 20461 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42570
I20260924 15:59:37.850065 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20523
Sep 24 15:59:37 dist-test-slave-xn4h krb5kdc[20416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265577, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:37.866534 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:37.851558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.16.233.129:45447)
0924 15:59:37.852223 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:37.852235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:37.852268 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:59:37.852358 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:37.852369 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:37.852697 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:37.852858 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:37.854688 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.855750 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.856881 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.857441 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.857537 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:37.861284 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:37.861296 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:37.861310 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:37.861343 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:37.864163 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.864791 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.864797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.864800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.864922 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.865232 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.865235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.865237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.865557 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:37.865790 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:37.865982 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 15:59:37.866177 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":345,"threads_started":1}
W20260924 15:59:37.867815 20648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58494
W20260924 15:59:37.868965 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20523: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 15:59:37.869160 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 20523
I20260924 15:59:37.885511 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20659
Sep 24 15:59:37 dist-test-slave-xn4h krb5kdc[20416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265577, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 15:59:37.901652 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:37.886531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.16.233.130:39463)
0924 15:59:37.887051 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:37.887059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:37.887086 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:59:37.887150 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:37.887158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:37.887384 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:37.887525 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:37.888731 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.889760 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.891070 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.891637 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.891749 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:37.895824 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:37.895839 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:37.895855 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:37.895901 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:37.899050 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.899721 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.899729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.899731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.899862 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.900190 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.900193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.900194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.900553 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:37.900809 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:37.901044 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 15:59:37.901249 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
W20260924 15:59:37.903009 20783 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55104
W20260924 15:59:37.904175 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20659: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 15:59:37.904320 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 20659
I20260924 15:59:37.920261 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20793
Sep 24 15:59:37 dist-test-slave-xn4h krb5kdc[20416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265577, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 15:59:37.936579 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:37.921371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.16.233.131:39033)
0924 15:59:37.921999 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:37.922006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:37.922034 (+ 28us) server_negotiation.cc:407] Connection header received
0924 15:59:37.922094 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:37.922101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:37.922366 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:37.922530 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:37.923860 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.924913 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.926181 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:37.926817 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:37.926954 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:37.931221 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:37.931232 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:37.931247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:37.931282 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:37.933971 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.934739 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.934747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.934749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.934866 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:37.935245 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:37.935249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:37.935251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:37.935584 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:37.935819 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:37.936078 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 15:59:37.936280 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":341,"threads_started":1}
W20260924 15:59:37.937947 20917 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41306
W20260924 15:59:37.939097 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20793: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 15:59:37.939275 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 20793
I20260924 15:59:37.955137 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20429
I20260924 15:59:37.956284 20491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:38.059705 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 20429
2026-09-24T15:59:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3639 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:38 dist-test-slave-xn4h krb5kdc[20939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:38 dist-test-slave-xn4h krb5kdc[20939](info): set up 2 sockets
Sep 24 15:59:38 dist-test-slave-xn4h krb5kdc[20939](info): commencing operation
krb5kdc: starting...
W20260924 15:59:39.164467 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:39 dist-test-slave-xn4h krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:39.370029 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:38859
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:38859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:39.706991 20952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:39.707410 20952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:39.707587 20952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:39.715238 20952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:39.715395 20952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:39.715510 20952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:39.715615 20952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:39.715705 20952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:39.725907 20952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:38859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:38859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:39.727361 20952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:39.728778 20952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:39.740442 20957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:39.740412 20958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:39.740512 20961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:39.741731 20952 server_base.cc:1034] running on GCE node
I20260924 15:59:39.742722 20952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:39.744262 20952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:39.745512 20952 hybrid_clock.cc:648] HybridClock initialized: now 1790265579745444 us; error 56 us; skew 500 ppm
I20260924 15:59:39.745903 20952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:39 dist-test-slave-xn4h krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265579, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:39.749117 20952 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:39.750706 20952 webserver.cc:460] Webserver started at http://127.16.233.190:35249/ using document root <none> and password file <none>
I20260924 15:59:39.751433 20952 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:39.751582 20952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:39.751911 20952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:39.755029 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "a3a8f317b18b4a5d8fb18d2520a9de80"
format_stamp: "Formatted at 2026-09-24 15:59:39 on dist-test-slave-xn4h"
server_key: "5240cdab686d01aa03229ab2420e06bf"
server_key_iv: "8108d662ce4842f9551d548299864561"
server_key_version: "encryptionkey@0"
I20260924 15:59:39.755911 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "a3a8f317b18b4a5d8fb18d2520a9de80"
format_stamp: "Formatted at 2026-09-24 15:59:39 on dist-test-slave-xn4h"
server_key: "5240cdab686d01aa03229ab2420e06bf"
server_key_iv: "8108d662ce4842f9551d548299864561"
server_key_version: "encryptionkey@0"
I20260924 15:59:39.761158 20952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 15:59:39.764717 20968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:39.766109 20952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 15:59:39.766345 20952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "a3a8f317b18b4a5d8fb18d2520a9de80"
format_stamp: "Formatted at 2026-09-24 15:59:39 on dist-test-slave-xn4h"
server_key: "5240cdab686d01aa03229ab2420e06bf"
server_key_iv: "8108d662ce4842f9551d548299864561"
server_key_version: "encryptionkey@0"
I20260924 15:59:39.766564 20952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:39.798629 20952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:39.803320 20952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:39.803623 20952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:39.821949 20952 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:38859
I20260924 15:59:39.821980 21019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:38859 every 8 connection(s)
I20260924 15:59:39.823442 20952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:39.827193 21020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:39.830780 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 20952
I20260924 15:59:39.831017 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:39.831813 17318 external_mini_cluster.cc:1428] Setting key 786ae78142472b802908b09868242c95
I20260924 15:59:39.840772 21020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80: Bootstrap starting.
I20260924 15:59:39.844864 21020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:39.846266 21020 log.cc:824] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80: Log is configured to *not* fsync() on all Append() calls
Sep 24 15:59:39 dist-test-slave-xn4h krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:39.849463 21020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80: No bootstrap required, opened a new log
I20260924 15:59:39.855755 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:39.833942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.16.233.190:38859)
0924 15:59:39.834771 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:39.834783 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:39.834825 (+ 42us) server_negotiation.cc:407] Connection header received
0924 15:59:39.836495 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:39.836521 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:39.836940 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:39.837301 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:39.839189 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:39.840520 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:39.842245 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:39.843009 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:39.843144 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:39.848145 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:39.848153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:39.848170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:39.848209 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:39.851447 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:39.852196 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:39.852204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:39.852211 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:39.852389 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:39.852784 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:39.852790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:39.852802 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:39.853364 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:39.853699 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:39.854137 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 15:59:39.854613 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":399,"threads_started":1}
I20260924 15:59:39.858125 21020 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a8f317b18b4a5d8fb18d2520a9de80" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38859 } }
I20260924 15:59:39.858578 21020 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:39.858711 21020 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a8f317b18b4a5d8fb18d2520a9de80, State: Initialized, Role: FOLLOWER
I20260924 15:59:39.859468 21020 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a8f317b18b4a5d8fb18d2520a9de80" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38859 } }
I20260924 15:59:39.859783 21020 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:39.859946 21020 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:39.860126 21020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:39.862988 21020 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a8f317b18b4a5d8fb18d2520a9de80" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38859 } }
I20260924 15:59:39.863515 21020 leader_election.cc:302] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3a8f317b18b4a5d8fb18d2520a9de80; no voters:
I20260924 15:59:39.864179 21020 leader_election.cc:288] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:39.864367 21025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:39.865631 21025 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [term 1 LEADER]: Becoming Leader. State: Replica: a3a8f317b18b4a5d8fb18d2520a9de80, State: Running, Role: LEADER
I20260924 15:59:39.866254 21025 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a8f317b18b4a5d8fb18d2520a9de80" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38859 } }
I20260924 15:59:39.866600 21020 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:39.868813 21027 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3a8f317b18b4a5d8fb18d2520a9de80. Latest consensus state: current_term: 1 leader_uuid: "a3a8f317b18b4a5d8fb18d2520a9de80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a8f317b18b4a5d8fb18d2520a9de80" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38859 } } }
I20260924 15:59:39.869172 21027 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:39.869663 21026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3a8f317b18b4a5d8fb18d2520a9de80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a8f317b18b4a5d8fb18d2520a9de80" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38859 } } }
I20260924 15:59:39.869932 21026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:39.872067 21032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:39.878285 21032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:39.888010 21032 catalog_manager.cc:1347] Generated new cluster ID: 3de8724feffb4139a7e348d7d47319c1
I20260924 15:59:39.888135 21032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:39.897594 21032 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:39.899549 21032 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:39.922811 21032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3a8f317b18b4a5d8fb18d2520a9de80: Generated new TSK 0
I20260924 15:59:39.924007 21032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 15:59:39.977869 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38859
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:40.312690 21046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:40.312987 21046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:40.313081 21046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:40.320413 21046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:40.320529 21046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:40.320644 21046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:40.331210 21046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:40.333084 21046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:40.334455 21046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:40.347622 21051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:40.347656 21054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:40.347695 21052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:40.348549 21046 server_base.cc:1034] running on GCE node
I20260924 15:59:40.349162 21046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:40.350101 21046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:40.351336 21046 hybrid_clock.cc:648] HybridClock initialized: now 1790265580351293 us; error 57 us; skew 500 ppm
I20260924 15:59:40.351656 21046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:40 dist-test-slave-xn4h krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:40.354745 21046 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 15:59:40.356240 21046 webserver.cc:460] Webserver started at http://127.16.233.129:45565/ using document root <none> and password file <none>
I20260924 15:59:40.356997 21046 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:40.357146 21046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:40.357461 21046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:40.360395 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "5e6e1e2d9ec6421f97f015a3bcc8be8a"
format_stamp: "Formatted at 2026-09-24 15:59:40 on dist-test-slave-xn4h"
server_key: "d0c30aaa75333f61fb95ab39d15f4fe1"
server_key_iv: "55baa95361726e62d91a9bf681f5973e"
server_key_version: "encryptionkey@0"
I20260924 15:59:40.361189 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "5e6e1e2d9ec6421f97f015a3bcc8be8a"
format_stamp: "Formatted at 2026-09-24 15:59:40 on dist-test-slave-xn4h"
server_key: "d0c30aaa75333f61fb95ab39d15f4fe1"
server_key_iv: "55baa95361726e62d91a9bf681f5973e"
server_key_version: "encryptionkey@0"
I20260924 15:59:40.366500 21046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 15:59:40.370517 21061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:40.371836 21046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:40.372009 21046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "5e6e1e2d9ec6421f97f015a3bcc8be8a"
format_stamp: "Formatted at 2026-09-24 15:59:40 on dist-test-slave-xn4h"
server_key: "d0c30aaa75333f61fb95ab39d15f4fe1"
server_key_iv: "55baa95361726e62d91a9bf681f5973e"
server_key_version: "encryptionkey@0"
I20260924 15:59:40.372234 21046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:40.415858 21046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:40.419718 21046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:40.420023 21046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:40.421000 21046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:40.422351 21046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:40.422456 21046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:40.422606 21046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:40.422683 21046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:40.455060 21046 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:33369
I20260924 15:59:40.455107 21174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:33369 every 8 connection(s)
I20260924 15:59:40.456357 21046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:40.459698 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 21046
I20260924 15:59:40.459918 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:40.460772 17318 external_mini_cluster.cc:1428] Setting key fae920805f19154bd1bf8113fb7565cb
Sep 24 15:59:40 dist-test-slave-xn4h krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:40.480384 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:40.459058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:58769 (local address 127.16.233.190:38859)
0924 15:59:40.459541 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:40.459547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:40.460780 (+ 1233us) server_negotiation.cc:407] Connection header received
0924 15:59:40.463065 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:40.463069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:40.463131 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:40.463239 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:40.465365 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:40.466064 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:40.468622 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:40.469352 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:40.469495 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:40.473048 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:40.473057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:40.473065 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:40.473110 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:40.476808 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:40.477572 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:40.477579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:40.477586 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:40.477743 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:40.478191 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:40.478199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:40.478205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:40.478625 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:40.478791 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:40.479661 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 15:59:40.479916 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 15:59:40.481137 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:40.459694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:38859 (local address 127.16.233.129:58769)
0924 15:59:40.460602 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:40.460655 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:40.462777 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:40.463506 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:40.463517 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:40.464306 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:40.465175 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:40.465205 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:40.466244 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:40.466253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:40.468452 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:40.468470 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:40.469601 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:40.469606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:40.469852 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:40.470629 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:40.470662 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:40.472834 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:40.476943 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:40.476967 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:40.476986 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:40.477431 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:40.477880 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:40.477883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:40.477886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:40.478061 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:40.478769 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:40.478777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:40.479182 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 15:59:40.479519 (+ 337us) client_negotiation.cc:240] Negotiation successful
0924 15:59:40.479992 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":360,"threads_started":1}
I20260924 15:59:40.482800 21175 heartbeater.cc:347] Connected to a master server at 127.16.233.190:38859
I20260924 15:59:40.483155 21175 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
I20260924 15:59:40.483934 21175 heartbeater.cc:511] Master 127.16.233.190:38859 requested a full tablet report, sending...
I20260924 15:59:40.486217 20985 ts_manager.cc:194] Registered new tserver with Master: 5e6e1e2d9ec6421f97f015a3bcc8be8a (127.16.233.129:33369)
I20260924 15:59:40.488364 20985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:58769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 15:59:40.507047 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38859
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:40.825405 21181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:40.825735 21181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:40.825911 21181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:40.833577 21181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:40.833738 21181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:40.833913 21181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:40.844194 21181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:40.845635 21181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:40.847052 21181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:40.860782 21186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:40.860745 21187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:40.860944 21189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:40.862001 21181 server_base.cc:1034] running on GCE node
I20260924 15:59:40.862814 21181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:40.864143 21181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:40.865386 21181 hybrid_clock.cc:648] HybridClock initialized: now 1790265580865321 us; error 51 us; skew 500 ppm
I20260924 15:59:40.865731 21181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:40 dist-test-slave-xn4h krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:40.869014 21181 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 15:59:40.870749 21181 webserver.cc:460] Webserver started at http://127.16.233.130:39613/ using document root <none> and password file <none>
I20260924 15:59:40.871515 21181 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:40.871654 21181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:40.871963 21181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:40.875049 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "657509f1af854d069a6d02f5c593b54f"
format_stamp: "Formatted at 2026-09-24 15:59:40 on dist-test-slave-xn4h"
server_key: "4c473c03bd5bc50fa001e08ed9efa148"
server_key_iv: "0c01ac0fe1fe0b334389c6180e9619e4"
server_key_version: "encryptionkey@0"
I20260924 15:59:40.875865 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "657509f1af854d069a6d02f5c593b54f"
format_stamp: "Formatted at 2026-09-24 15:59:40 on dist-test-slave-xn4h"
server_key: "4c473c03bd5bc50fa001e08ed9efa148"
server_key_iv: "0c01ac0fe1fe0b334389c6180e9619e4"
server_key_version: "encryptionkey@0"
I20260924 15:59:40.881203 21181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 15:59:40.884907 21196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:40.886384 21181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 15:59:40.886561 21181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "657509f1af854d069a6d02f5c593b54f"
format_stamp: "Formatted at 2026-09-24 15:59:40 on dist-test-slave-xn4h"
server_key: "4c473c03bd5bc50fa001e08ed9efa148"
server_key_iv: "0c01ac0fe1fe0b334389c6180e9619e4"
server_key_version: "encryptionkey@0"
I20260924 15:59:40.886817 21181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:40.906983 21181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:40.911680 21181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:40.912006 21181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:40.913038 21181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:40.914602 21181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:40.914711 21181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:40.914872 21181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:40.914947 21181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:40.949054 21181 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:43243
I20260924 15:59:40.949098 21310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:43243 every 8 connection(s)
I20260924 15:59:40.950533 21181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:40.955495 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 21181
I20260924 15:59:40.955708 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:40.956233 17318 external_mini_cluster.cc:1428] Setting key 666d16299771ef258a2bcaa4f3c58b62
Sep 24 15:59:40 dist-test-slave-xn4h krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:40.975118 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:40.953162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:58401 (local address 127.16.233.190:38859)
0924 15:59:40.953354 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:40.953358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:40.954602 (+ 1244us) server_negotiation.cc:407] Connection header received
0924 15:59:40.957230 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:40.957236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:40.957336 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:40.957444 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:40.959568 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:40.960303 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:40.962903 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:40.963539 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:40.963679 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:40.967871 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:40.967880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:40.967888 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:40.967939 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:40.971648 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:40.972468 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:40.972477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:40.972483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:40.972648 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:40.973096 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:40.973104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:40.973110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:40.973570 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:40.973736 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:40.974683 (+ 947us) server_negotiation.cc:299] Negotiation successful
0924 15:59:40.974876 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 15:59:40.976855 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:40.953708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:38859 (local address 127.16.233.130:58401)
0924 15:59:40.954442 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:40.954486 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:40.956935 (+ 2449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:40.957695 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:40.957705 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:40.958518 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:40.959369 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:40.959397 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:40.960464 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:40.960473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:40.962727 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:40.962746 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:40.963793 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:40.963803 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:40.964154 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:40.964949 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:40.964978 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:40.967498 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:40.971822 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:40.971830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:40.971845 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:40.972318 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:40.972758 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:40.972761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:40.972763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:40.972960 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:40.973709 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:40.973716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:40.974203 (+ 487us) client_negotiation.cc:769] Sending connection context
0924 15:59:40.974587 (+ 384us) client_negotiation.cc:240] Negotiation successful
0924 15:59:40.975259 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":262,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 15:59:40.978441 21311 heartbeater.cc:347] Connected to a master server at 127.16.233.190:38859
I20260924 15:59:40.978824 21311 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:40.979637 21311 heartbeater.cc:511] Master 127.16.233.190:38859 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
I20260924 15:59:40.981343 20985 ts_manager.cc:194] Registered new tserver with Master: 657509f1af854d069a6d02f5c593b54f (127.16.233.130:43243)
I20260924 15:59:40.982342 20985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:58401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 15:59:41.005470 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38859
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:41.326539 21316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:41.326814 21316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:41.326972 21316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:41.334378 21316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:41.334507 21316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:41.334697 21316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:41.344727 21316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:41.346065 21316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:41.347410 21316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:41.360469 21321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:41.360489 21324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:41.360589 21322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:41.363600 21316 server_base.cc:1034] running on GCE node
I20260924 15:59:41.364202 21316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:41.365165 21316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:41.366437 21316 hybrid_clock.cc:648] HybridClock initialized: now 1790265581366378 us; error 40 us; skew 500 ppm
I20260924 15:59:41.366775 21316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:41 dist-test-slave-xn4h krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265581, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:41.369777 21316 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 15:59:41.371280 21316 webserver.cc:460] Webserver started at http://127.16.233.131:44615/ using document root <none> and password file <none>
I20260924 15:59:41.371929 21316 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:41.372028 21316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:41.372267 21316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:41.375129 21316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "a7924ac683694b51af7e2a66ad3e56fa"
format_stamp: "Formatted at 2026-09-24 15:59:41 on dist-test-slave-xn4h"
server_key: "580098bb9ba279bf6eebacd20f5dd73f"
server_key_iv: "f1db96f446dbc74bf50007b4d9f3f841"
server_key_version: "encryptionkey@0"
I20260924 15:59:41.375909 21316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "a7924ac683694b51af7e2a66ad3e56fa"
format_stamp: "Formatted at 2026-09-24 15:59:41 on dist-test-slave-xn4h"
server_key: "580098bb9ba279bf6eebacd20f5dd73f"
server_key_iv: "f1db96f446dbc74bf50007b4d9f3f841"
server_key_version: "encryptionkey@0"
I20260924 15:59:41.380950 21316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 15:59:41.384166 21331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:41.385403 21316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 15:59:41.385564 21316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "a7924ac683694b51af7e2a66ad3e56fa"
format_stamp: "Formatted at 2026-09-24 15:59:41 on dist-test-slave-xn4h"
server_key: "580098bb9ba279bf6eebacd20f5dd73f"
server_key_iv: "f1db96f446dbc74bf50007b4d9f3f841"
server_key_version: "encryptionkey@0"
I20260924 15:59:41.385761 21316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:41.422478 21316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:41.426568 21316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:41.426883 21316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:41.427847 21316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:41.429200 21316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:41.429273 21316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:41.429448 21316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:41.429498 21316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:41.463395 21316 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:33197
I20260924 15:59:41.463439 21447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:33197 every 8 connection(s)
I20260924 15:59:41.464649 21316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:41.474588 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 21316
I20260924 15:59:41.474845 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:41.475303 17318 external_mini_cluster.cc:1428] Setting key 722ab291b188539544c186f82577fd15
Sep 24 15:59:41 dist-test-slave-xn4h krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265581, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:41.486135 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:41.467284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:51459 (local address 127.16.233.190:38859)
0924 15:59:41.467570 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:41.467577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:41.468736 (+ 1159us) server_negotiation.cc:407] Connection header received
0924 15:59:41.470688 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:41.470696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:41.470778 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:41.470864 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:41.472379 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.473169 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.475031 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.475497 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.475594 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:41.479839 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:41.479845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:41.479848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:41.479881 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:41.482846 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.483825 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.483831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.483835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.483974 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.484409 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.484414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.484417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.484681 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:41.484837 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:41.485553 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 15:59:41.485779 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260924 15:59:41.487159 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:41.467909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:38859 (local address 127.16.233.131:51459)
0924 15:59:41.468584 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:41.468621 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:41.470407 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:41.471110 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:41.471118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:41.471651 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:41.472196 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:41.472213 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.473296 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.473299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:41.474892 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:41.474909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.475730 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.475738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:41.476079 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:41.476815 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:41.476854 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:41.479641 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:41.482974 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:41.482984 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:41.483004 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:41.483698 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:41.484085 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:41.484089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:41.484093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:41.484289 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:41.484815 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:41.484823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:41.485184 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 15:59:41.485456 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 15:59:41.486004 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":263,"threads_started":1}
I20260924 15:59:41.488571 21448 heartbeater.cc:347] Connected to a master server at 127.16.233.190:38859
I20260924 15:59:41.488934 21448 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:41.489894 21448 heartbeater.cc:511] Master 127.16.233.190:38859 requested a full tablet report, sending...
I20260924 15:59:41.491324 21175 heartbeater.cc:503] Master 127.16.233.190:38859 was elected leader, sending a full tablet report...
I20260924 15:59:41.491676 20985 ts_manager.cc:194] Registered new tserver with Master: a7924ac683694b51af7e2a66ad3e56fa (127.16.233.131:33197)
I20260924 15:59:41.492813 20985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:51459
I20260924 15:59:41.503386 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:59:41 dist-test-slave-xn4h krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:59:41 dist-test-slave-xn4h krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:41.551554 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:41.536665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.16.233.129:33369)
0924 15:59:41.537259 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:41.537266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:41.537293 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:59:41.537354 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:41.537361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:41.537571 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:41.537730 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:41.539239 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.540261 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.541406 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.541970 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.542073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:41.546109 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:41.546122 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:41.546136 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:41.546172 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:41.548941 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.549575 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.549583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.549585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.549732 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.550061 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.550065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.550067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.550503 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:41.550752 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:41.550966 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 15:59:41.551166 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":287,"threads_started":1}
Sep 24 15:59:41 dist-test-slave-xn4h krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:41.569375 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:41.555753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.16.233.190:38859)
0924 15:59:41.555982 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:41.555991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:41.556121 (+ 130us) server_negotiation.cc:407] Connection header received
0924 15:59:41.556250 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:41.556254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:41.556319 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:41.556400 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:41.557584 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.558422 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.559771 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.560205 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.560300 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:41.564114 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:41.564119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:41.564122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:41.564156 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:41.566736 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.567384 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.567388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.567390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.567521 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.567910 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.567916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.567920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.568290 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:41.568554 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:41.568846 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 15:59:41.569059 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 15:59:41.569988 20985 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42196
I20260924 15:59:41.571069 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 21046
I20260924 15:59:41.583302 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:41.572097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.16.233.129:33369)
0924 15:59:41.572280 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:41.572284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:41.572378 (+ 94us) server_negotiation.cc:407] Connection header received
0924 15:59:41.572498 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:41.572502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:41.572592 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:41.572681 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:41.573776 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.574518 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.575790 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.576291 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.576435 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:41.578683 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:41.578688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:41.578691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:41.578722 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:41.581179 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.581743 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.581746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.581748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.581859 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.582147 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.582150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.582152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.582484 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:41.582652 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:41.582900 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 15:59:41.583053 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 15:59:41.583933 21170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:41.666225 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 21046
I20260924 15:59:41.682904 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 21181
Sep 24 15:59:41 dist-test-slave-xn4h krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 15:59:41.710692 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:41.683898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.16.233.130:43243)
0924 15:59:41.684424 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:41.684432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:41.684459 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:59:41.684523 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:41.684528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:41.684755 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:41.684883 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:41.685971 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.687031 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.688222 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.688802 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.688937 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:41.693400 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:41.693413 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:41.693428 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:41.693466 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:41.707643 (+ 14177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.708439 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.708448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.708450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.708586 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.709011 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.709018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.709026 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.709388 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:41.709729 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:41.710039 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 15:59:41.710349 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":272,"threads_started":1}
I20260924 15:59:41.712016 21306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:41.793848 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 21181
I20260924 15:59:41.809957 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 21316
Sep 24 15:59:41 dist-test-slave-xn4h krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 15:59:41.827148 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:41.811077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.16.233.131:33197)
0924 15:59:41.811732 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:41.811739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:41.811768 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:59:41.811827 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:41.811833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:41.812040 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:41.812270 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:41.813573 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.814740 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.816078 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:41.816680 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:41.816828 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:41.821127 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:41.821139 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:41.821156 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:41.821198 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:41.824335 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.825017 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.825024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.825026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.825155 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:41.825531 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:41.825536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:41.825539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:41.825893 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:41.826217 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:41.826525 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 15:59:41.826808 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":347,"threads_started":1}
I20260924 15:59:41.828428 21443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:41.912824 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 21316
I20260924 15:59:41.930847 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 20952
I20260924 15:59:41.932120 21015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:42.035506 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 20952
2026-09-24T15:59:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3974 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:42 dist-test-slave-xn4h krb5kdc[21467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:42 dist-test-slave-xn4h krb5kdc[21467](info): set up 2 sockets
Sep 24 15:59:42 dist-test-slave-xn4h krb5kdc[21467](info): commencing operation
krb5kdc: starting...
W20260924 15:59:43.132133 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:43 dist-test-slave-xn4h krb5kdc[21467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:43.333355 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:37225
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:37225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:43.654109 21480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:43.654443 21480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:43.654610 21480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:43.661664 21480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:43.661772 21480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:43.661895 21480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:43.662009 21480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:43.662086 21480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:43.671933 21480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:37225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:37225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:43.673230 21480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:43.674546 21480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:43.686030 21486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:43.686089 21489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:43.686010 21487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:43.686900 21480 server_base.cc:1034] running on GCE node
I20260924 15:59:43.687578 21480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:43.688838 21480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:43.690034 21480 hybrid_clock.cc:648] HybridClock initialized: now 1790265583689971 us; error 52 us; skew 500 ppm
I20260924 15:59:43.690375 21480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:43 dist-test-slave-xn4h krb5kdc[21467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:43.693419 21480 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:43.694629 21480 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:43.694756 21480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:43.695077 21480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:43.698010 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "a8b50d9712784862bec56997d3dca1b4"
format_stamp: "Formatted at 2026-09-24 15:59:43 on dist-test-slave-xn4h"
server_key: "caac286eefcc8e761b8d70c0e4d325da"
server_key_iv: "e7867e7f77ff279398420f1a45d9b744"
server_key_version: "encryptionkey@0"
I20260924 15:59:43.698830 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "a8b50d9712784862bec56997d3dca1b4"
format_stamp: "Formatted at 2026-09-24 15:59:43 on dist-test-slave-xn4h"
server_key: "caac286eefcc8e761b8d70c0e4d325da"
server_key_iv: "e7867e7f77ff279398420f1a45d9b744"
server_key_version: "encryptionkey@0"
I20260924 15:59:43.703781 21480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 15:59:43.707109 21495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:43.708329 21480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:43.708511 21480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "a8b50d9712784862bec56997d3dca1b4"
format_stamp: "Formatted at 2026-09-24 15:59:43 on dist-test-slave-xn4h"
server_key: "caac286eefcc8e761b8d70c0e4d325da"
server_key_iv: "e7867e7f77ff279398420f1a45d9b744"
server_key_version: "encryptionkey@0"
I20260924 15:59:43.708722 21480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:43.724769 21480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:43.728796 21480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:43.729018 21480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:43.746174 21480 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:37225
I20260924 15:59:43.746196 21547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:37225 every 8 connection(s)
I20260924 15:59:43.747113 21480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:43.750916 21548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:43.751519 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 21480
I20260924 15:59:43.751752 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:43.752391 17318 external_mini_cluster.cc:1428] Setting key e0860244c5e6a45c31a75aeacef90ff0
I20260924 15:59:43.764909 21548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4: Bootstrap starting.
Sep 24 15:59:43 dist-test-slave-xn4h krb5kdc[21467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:43.768631 21548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:43.769871 21548 log.cc:824] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:43.772775 21548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4: No bootstrap required, opened a new log
I20260924 15:59:43.776402 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:43.754436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.16.233.190:37225)
0924 15:59:43.755315 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:43.755327 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:43.755374 (+ 47us) server_negotiation.cc:407] Connection header received
0924 15:59:43.757007 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:43.757030 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:43.757444 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:43.757782 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:43.759650 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:43.760925 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:43.762752 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:43.763421 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:43.763549 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:43.768223 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:43.768231 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:43.768245 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:43.768283 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:43.771736 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:43.772425 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:43.772431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:43.772437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:43.772599 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:43.773033 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:43.773038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:43.773041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:43.773813 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:43.774123 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:43.774479 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 15:59:43.774972 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":434,"threads_started":1}
I20260924 15:59:43.781991 21548 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50d9712784862bec56997d3dca1b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37225 } }
I20260924 15:59:43.782382 21548 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:43.782466 21548 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b50d9712784862bec56997d3dca1b4, State: Initialized, Role: FOLLOWER
I20260924 15:59:43.782953 21548 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50d9712784862bec56997d3dca1b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37225 } }
I20260924 15:59:43.783149 21548 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:43.783241 21548 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:43.783368 21548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:43.785465 21548 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50d9712784862bec56997d3dca1b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37225 } }
I20260924 15:59:43.786101 21548 leader_election.cc:302] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8b50d9712784862bec56997d3dca1b4; no voters:
I20260924 15:59:43.787002 21548 leader_election.cc:288] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:43.787132 21553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:43.787849 21553 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [term 1 LEADER]: Becoming Leader. State: Replica: a8b50d9712784862bec56997d3dca1b4, State: Running, Role: LEADER
I20260924 15:59:43.788332 21553 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50d9712784862bec56997d3dca1b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37225 } }
I20260924 15:59:43.789045 21548 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:43.791204 21555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8b50d9712784862bec56997d3dca1b4. Latest consensus state: current_term: 1 leader_uuid: "a8b50d9712784862bec56997d3dca1b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50d9712784862bec56997d3dca1b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37225 } } }
I20260924 15:59:43.791611 21555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:43.791867 21554 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8b50d9712784862bec56997d3dca1b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50d9712784862bec56997d3dca1b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37225 } } }
I20260924 15:59:43.792196 21554 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:43.794040 21559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:43.799729 21559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:43.809499 21559 catalog_manager.cc:1347] Generated new cluster ID: 4feda9373d16421bb7737dd0cfc6e331
I20260924 15:59:43.809628 21559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:43.823222 21559 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:43.824569 21559 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:43.833101 21559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4: Generated new TSK 0
I20260924 15:59:43.833945 21559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 15:59:43.891069 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37225
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:44.247678 21574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:44.247967 21574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:44.248066 21574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:44.255627 21574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:44.255743 21574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:44.255859 21574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:44.265856 21574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:44.267184 21574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:44.268460 21574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:44.281344 21582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:44.281572 21580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:44.281883 21579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:44.282459 21574 server_base.cc:1034] running on GCE node
I20260924 15:59:44.283116 21574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:44.284078 21574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:44.285310 21574 hybrid_clock.cc:648] HybridClock initialized: now 1790265584285250 us; error 47 us; skew 500 ppm
I20260924 15:59:44.285645 21574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:44 dist-test-slave-xn4h krb5kdc[21467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:44.288782 21574 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 15:59:44.289976 21574 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:44.290110 21574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:44.290517 21574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:44.293452 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "a57035562f344e8885e3ddcd5763de81"
format_stamp: "Formatted at 2026-09-24 15:59:44 on dist-test-slave-xn4h"
server_key: "91f737fed79126bd03d3664c2cfe8c72"
server_key_iv: "2e29ac46dc5b971ef2531c659faa8ac3"
server_key_version: "encryptionkey@0"
I20260924 15:59:44.294489 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "a57035562f344e8885e3ddcd5763de81"
format_stamp: "Formatted at 2026-09-24 15:59:44 on dist-test-slave-xn4h"
server_key: "91f737fed79126bd03d3664c2cfe8c72"
server_key_iv: "2e29ac46dc5b971ef2531c659faa8ac3"
server_key_version: "encryptionkey@0"
I20260924 15:59:44.299893 21574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 15:59:44.303359 21588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:44.304715 21574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 15:59:44.304891 21574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "a57035562f344e8885e3ddcd5763de81"
format_stamp: "Formatted at 2026-09-24 15:59:44 on dist-test-slave-xn4h"
server_key: "91f737fed79126bd03d3664c2cfe8c72"
server_key_iv: "2e29ac46dc5b971ef2531c659faa8ac3"
server_key_version: "encryptionkey@0"
I20260924 15:59:44.305115 21574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:44.336268 21574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:44.340948 21574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:44.341256 21574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:44.342239 21574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:44.343748 21574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:44.343837 21574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:44.343915 21574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:44.343972 21574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:44.376691 21574 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:33313
I20260924 15:59:44.376761 21702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:33313 every 8 connection(s)
I20260924 15:59:44.377564 21574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:44.383898 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 21574
I20260924 15:59:44.384137 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:44.384784 17318 external_mini_cluster.cc:1428] Setting key bbdd1dd4fdbb0c9729f94c6606d4a658
Sep 24 15:59:44 dist-test-slave-xn4h krb5kdc[21467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:44.399000 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:44.380086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:43259 (local address 127.16.233.190:37225)
0924 15:59:44.380516 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:44.380520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:44.381504 (+ 984us) server_negotiation.cc:407] Connection header received
0924 15:59:44.383355 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:44.383361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:44.383467 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:44.383601 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:44.385300 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:44.386140 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:44.387894 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:44.388296 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:44.388435 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:44.392759 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:44.392764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:44.392768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:44.392803 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:44.395495 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:44.396504 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:44.396508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:44.396511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:44.396639 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:44.397141 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:44.397144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:44.397146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:44.397418 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:44.397529 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:44.398532 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0924 15:59:44.398720 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260924 15:59:44.400628 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:44.380678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:37225 (local address 127.16.233.129:43259)
0924 15:59:44.381350 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:44.381387 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:44.383083 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:44.383838 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:44.383846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:44.384498 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:44.385120 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:44.385141 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:44.386344 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:44.386351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:44.387732 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:44.387745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:44.388523 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:44.388531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:44.388763 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:44.389630 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:44.389660 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:44.392462 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:44.395647 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:44.395660 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:44.395682 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:44.396375 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:44.396757 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:44.396765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:44.396772 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:44.397027 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:44.397550 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:44.397562 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:44.398071 (+ 509us) client_negotiation.cc:769] Sending connection context
0924 15:59:44.398427 (+ 356us) client_negotiation.cc:240] Negotiation successful
0924 15:59:44.399049 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":272,"threads_started":1}
I20260924 15:59:44.402493 21703 heartbeater.cc:347] Connected to a master server at 127.16.233.190:37225
I20260924 15:59:44.402848 21703 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:44.403548 21703 heartbeater.cc:511] Master 127.16.233.190:37225 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 15:59:44.405800 21513 ts_manager.cc:194] Registered new tserver with Master: a57035562f344e8885e3ddcd5763de81 (127.16.233.129:33313)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
I20260924 15:59:44.407588 21513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:43259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 15:59:44.434106 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37225
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:44.762638 21709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:44.762907 21709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:44.763002 21709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:44.770365 21709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:44.770484 21709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:44.770602 21709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:44.780401 21709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:44.781682 21709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:44.783068 21709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:44.796142 21714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:44.796231 21715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:44.796154 21717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:44.797163 21709 server_base.cc:1034] running on GCE node
I20260924 15:59:44.797734 21709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:44.798719 21709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:44.799914 21709 hybrid_clock.cc:648] HybridClock initialized: now 1790265584799854 us; error 51 us; skew 500 ppm
I20260924 15:59:44.800230 21709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:44 dist-test-slave-xn4h krb5kdc[21467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:44.803479 21709 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 15:59:44.804581 21709 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:44.804669 21709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:44.804937 21709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:44.807916 21709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "9c1d88edfc7745899396481faee4e5da"
format_stamp: "Formatted at 2026-09-24 15:59:44 on dist-test-slave-xn4h"
server_key: "7826b0284fbe585b6d010f675df423d3"
server_key_iv: "25890aa249b41e1a3f3b6d972b983b5e"
server_key_version: "encryptionkey@0"
I20260924 15:59:44.808802 21709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "9c1d88edfc7745899396481faee4e5da"
format_stamp: "Formatted at 2026-09-24 15:59:44 on dist-test-slave-xn4h"
server_key: "7826b0284fbe585b6d010f675df423d3"
server_key_iv: "25890aa249b41e1a3f3b6d972b983b5e"
server_key_version: "encryptionkey@0"
I20260924 15:59:44.814257 21709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 15:59:44.817586 21723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:44.818969 21709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:44.819156 21709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "9c1d88edfc7745899396481faee4e5da"
format_stamp: "Formatted at 2026-09-24 15:59:44 on dist-test-slave-xn4h"
server_key: "7826b0284fbe585b6d010f675df423d3"
server_key_iv: "25890aa249b41e1a3f3b6d972b983b5e"
server_key_version: "encryptionkey@0"
I20260924 15:59:44.819387 21709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:44.851341 21709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:44.856019 21709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:44.856338 21709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:44.857348 21709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:44.858774 21709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:44.858903 21709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:44.859043 21709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:44.859135 21709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:44.896171 21709 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:43665
I20260924 15:59:44.896217 21837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:43665 every 8 connection(s)
I20260924 15:59:44.897082 21709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:44.904240 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 21709
I20260924 15:59:44.904469 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:44.905028 17318 external_mini_cluster.cc:1428] Setting key 520c9a0265947271472b254d77de09f9
Sep 24 15:59:44 dist-test-slave-xn4h krb5kdc[21467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:44.920915 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:44.899734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:42595 (local address 127.16.233.190:37225)
0924 15:59:44.900233 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:44.900240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:44.901206 (+ 966us) server_negotiation.cc:407] Connection header received
0924 15:59:44.903067 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:44.903072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:44.903148 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:44.903251 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:44.905327 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:44.906150 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:44.908172 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:44.909014 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:44.909167 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:44.913420 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:44.913429 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:44.913435 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:44.913487 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:44.917195 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:44.918035 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:44.918044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:44.918050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:44.918227 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:44.918740 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:44.918748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:44.918755 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:44.919164 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:44.919328 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:44.920066 (+ 738us) server_negotiation.cc:299] Negotiation successful
0924 15:59:44.920361 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":236,"threads_started":1}
I20260924 15:59:44.921514 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:44.900341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:37225 (local address 127.16.233.130:42595)
0924 15:59:44.901038 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:44.901081 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:44.902781 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:44.903469 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:44.903477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:44.904183 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:44.905058 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:44.905095 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:44.906334 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:44.906339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:44.908008 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:44.908022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:44.909289 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:44.909298 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:44.909598 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:44.910433 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:44.910458 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:44.913157 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:44.917377 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:44.917385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:44.917401 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:44.917885 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:44.918403 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:44.918406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:44.918408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:44.918592 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:44.919290 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:44.919296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:44.919645 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 15:59:44.919895 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 15:59:44.920334 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":273,"threads_started":1}
I20260924 15:59:44.923282 21838 heartbeater.cc:347] Connected to a master server at 127.16.233.190:37225
I20260924 15:59:44.923647 21838 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:44.924401 21838 heartbeater.cc:511] Master 127.16.233.190:37225 requested a full tablet report, sending...
I20260924 15:59:44.926029 21513 ts_manager.cc:194] Registered new tserver with Master: 9c1d88edfc7745899396481faee4e5da (127.16.233.130:43665)
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 15:59:44.927030 21513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:42595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 15:59:44.953579 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37225
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:45.280273 21844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:45.280553 21844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:45.280654 21844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:45.288327 21844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:45.288457 21844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:45.288574 21844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:45.298938 21844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:45.300284 21844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:45.301555 21844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:45.315337 21849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:45.315444 21850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:45.315448 21852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:45.316424 21844 server_base.cc:1034] running on GCE node
I20260924 15:59:45.317147 21844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:45.318133 21844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:45.319339 21844 hybrid_clock.cc:648] HybridClock initialized: now 1790265585319296 us; error 84 us; skew 500 ppm
I20260924 15:59:45.319639 21844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:45 dist-test-slave-xn4h krb5kdc[21467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265585, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:45.323046 21844 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 15:59:45.324398 21844 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:45.324519 21844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:45.324885 21844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:45.328179 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "82666662f11947dc9ff4aa39fdc574f6"
format_stamp: "Formatted at 2026-09-24 15:59:45 on dist-test-slave-xn4h"
server_key: "1e065dc47b45f554bdbe506eecfed7b6"
server_key_iv: "c4a1cbaa22360aabb3975224963d445c"
server_key_version: "encryptionkey@0"
I20260924 15:59:45.329066 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "82666662f11947dc9ff4aa39fdc574f6"
format_stamp: "Formatted at 2026-09-24 15:59:45 on dist-test-slave-xn4h"
server_key: "1e065dc47b45f554bdbe506eecfed7b6"
server_key_iv: "c4a1cbaa22360aabb3975224963d445c"
server_key_version: "encryptionkey@0"
I20260924 15:59:45.334663 21844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260924 15:59:45.338569 21858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:45.340058 21844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:45.340296 21844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "82666662f11947dc9ff4aa39fdc574f6"
format_stamp: "Formatted at 2026-09-24 15:59:45 on dist-test-slave-xn4h"
server_key: "1e065dc47b45f554bdbe506eecfed7b6"
server_key_iv: "c4a1cbaa22360aabb3975224963d445c"
server_key_version: "encryptionkey@0"
I20260924 15:59:45.340562 21844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:45.356721 21844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:45.361304 21844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:45.361606 21844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:45.362633 21844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:45.364115 21844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:45.364241 21844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:45.364382 21844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:45.364475 21844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:45.399978 21844 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:43805
I20260924 15:59:45.400025 21972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:43805 every 8 connection(s)
I20260924 15:59:45.400898 21844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:45.404403 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 21844
I20260924 15:59:45.404613 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:45.405128 17318 external_mini_cluster.cc:1428] Setting key 342c77ee516fdf7e97947a44c6d4fd9c
I20260924 15:59:45.410841 21703 heartbeater.cc:503] Master 127.16.233.190:37225 was elected leader, sending a full tablet report...
Sep 24 15:59:45 dist-test-slave-xn4h krb5kdc[21467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265585, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:45.422008 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.403422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:39861 (local address 127.16.233.190:37225)
0924 15:59:45.403614 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.403618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.404770 (+ 1152us) server_negotiation.cc:407] Connection header received
0924 15:59:45.406989 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.407015 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.407110 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.407215 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:45.409122 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.409948 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.411797 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.412373 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.412513 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.415891 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:45.415896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:45.415899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:45.415931 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:45.418709 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:45.419700 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:45.419704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:45.419706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:45.419872 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:45.420376 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:45.420381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:45.420384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:45.420761 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:45.420889 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.421582 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.421754 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 15:59:45.423362 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.403779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:37225 (local address 127.16.233.131:39861)
0924 15:59:45.404588 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:45.404640 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:45.406711 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:45.407515 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:45.407528 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:45.408321 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:45.408963 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.408984 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.410075 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.410080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.411653 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.411666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.412599 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.412605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.412854 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.413470 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:45.413493 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:45.415685 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:45.418882 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:45.418892 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:45.418912 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:45.419574 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:45.419993 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:45.419997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:45.420001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:45.420246 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:45.420876 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:45.420882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:45.421246 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 15:59:45.421481 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 15:59:45.421891 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":357,"threads_started":1}
I20260924 15:59:45.424796 21973 heartbeater.cc:347] Connected to a master server at 127.16.233.190:37225
I20260924 15:59:45.425123 21973 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:45.425796 21973 heartbeater.cc:511] Master 127.16.233.190:37225 requested a full tablet report, sending...
I20260924 15:59:45.427248 21513 ts_manager.cc:194] Registered new tserver with Master: 82666662f11947dc9ff4aa39fdc574f6 (127.16.233.131:43805)
I20260924 15:59:45.428051 21513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:39861
I20260924 15:59:45.432783 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:59:45.449510 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.437273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.16.233.190:37225)
0924 15:59:45.437463 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.437467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.437959 (+ 492us) server_negotiation.cc:407] Connection header received
0924 15:59:45.438098 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.438102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.438170 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.438251 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:45.439714 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.440462 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.441672 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.442076 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.442164 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.444428 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:45.444433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:45.444436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:45.444468 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:45.447147 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:45.447772 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:45.447775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:45.447777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:45.447897 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:45.448197 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:45.448200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:45.448202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:45.448519 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:45.448702 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.448965 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.449121 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 15:59:45.453720 21513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:59:45.460726 21513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:45.488664 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.473917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.16.233.129:33313)
0924 15:59:45.475371 (+ 1454us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.475380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.476067 (+ 687us) server_negotiation.cc:407] Connection header received
0924 15:59:45.476377 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.476388 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.476665 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.476884 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:45.478079 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.480258 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.484111 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.486345 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.486590 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.486865 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.487838 (+ 973us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.488118 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":1112,"threads_started":1}
I20260924 15:59:45.490078 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.473792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:33313 (local address 127.0.0.1:35228)
0924 15:59:45.475848 (+ 2056us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:45.475930 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:45.476126 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:45.477244 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:45.477256 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:45.477298 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:45.477853 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.477870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.480535 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.480544 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.483899 (+ 3355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.483915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.487102 (+ 3187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.487112 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.487443 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.487521 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 15:59:45.489336 (+ 1815us) client_negotiation.cc:240] Negotiation successful
0924 15:59:45.489630 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1866,"mutex_wait_us":29,"thread_start_us":2068,"threads_started":1}
I20260924 15:59:45.492061 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.474058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.16.233.130:43665)
0924 15:59:45.474852 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.474865 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.477803 (+ 2938us) server_negotiation.cc:407] Connection header received
0924 15:59:45.478175 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.478187 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.478510 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.478708 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:45.479870 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.482003 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.488709 (+ 6706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.490560 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.490736 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.490911 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.491550 (+ 639us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.491703 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":382,"threads_started":1}
I20260924 15:59:45.493062 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.473519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:43665 (local address 127.0.0.1:38498)
0924 15:59:45.475848 (+ 2329us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:45.477675 (+ 1827us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:45.477946 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:45.479057 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:45.479064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:45.479102 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:45.479630 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.479647 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.483807 (+ 4160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.483815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.488427 (+ 4612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.488453 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.491049 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.491056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.491263 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.491333 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 15:59:45.492515 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0924 15:59:45.492667 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1976,"mutex_wait_us":6,"thread_start_us":2000,"threads_started":1}
I20260924 15:59:45.494877 21638 tablet_service.cc:1467] Processing CreateTablet for tablet d5e875e20428422897bf804dd9bb1f24 (DEFAULT_TABLE table=test-table [id=7712e2b1bf1f4d11b77b9ca6aca35082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:45.496553 21638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e875e20428422897bf804dd9bb1f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:45.496656 21772 tablet_service.cc:1467] Processing CreateTablet for tablet d5e875e20428422897bf804dd9bb1f24 (DEFAULT_TABLE table=test-table [id=7712e2b1bf1f4d11b77b9ca6aca35082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:45.497884 21772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e875e20428422897bf804dd9bb1f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:45.498863 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.476372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:43805 (local address 127.0.0.1:58414)
0924 15:59:45.489786 (+ 13414us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:45.490391 (+ 605us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:45.490534 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:45.491186 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:45.491190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:45.491218 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:45.491622 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.491635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.493701 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.493709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.495682 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.495709 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.497932 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.497938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.498113 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.498146 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 15:59:45.498246 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 15:59:45.498418 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13265,"mutex_wait_us":43,"thread_start_us":13092,"threads_started":1}
I20260924 15:59:45.499816 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.476690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.16.233.131:43805)
0924 15:59:45.490289 (+ 13599us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.490298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.490681 (+ 383us) server_negotiation.cc:407] Connection header received
0924 15:59:45.490782 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.490788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.490995 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.491212 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:45.491881 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.493528 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.495907 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.497383 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.497577 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.497782 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.499258 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.499435 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13382,"thread_start_us":13206,"threads_started":1}
I20260924 15:59:45.506495 21907 tablet_service.cc:1467] Processing CreateTablet for tablet d5e875e20428422897bf804dd9bb1f24 (DEFAULT_TABLE table=test-table [id=7712e2b1bf1f4d11b77b9ca6aca35082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:45.508153 21907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e875e20428422897bf804dd9bb1f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:45.510146 21991 tablet_bootstrap.cc:492] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: Bootstrap starting.
I20260924 15:59:45.513700 21991 tablet_bootstrap.cc:654] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:45.514995 21991 log.cc:824] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:45.515188 21992 tablet_bootstrap.cc:492] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: Bootstrap starting.
I20260924 15:59:45.518594 21992 tablet_bootstrap.cc:654] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:45.518940 21991 tablet_bootstrap.cc:492] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: No bootstrap required, opened a new log
I20260924 15:59:45.519256 21991 ts_tablet_manager.cc:1397] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260924 15:59:45.519731 21992 log.cc:824] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:45.522473 21992 tablet_bootstrap.cc:492] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: No bootstrap required, opened a new log
I20260924 15:59:45.522758 21992 ts_tablet_manager.cc:1397] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 15:59:45.525437 21995 tablet_bootstrap.cc:492] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: Bootstrap starting.
I20260924 15:59:45.529304 21991 raft_consensus.cc:348] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } }
I20260924 15:59:45.529708 21991 raft_consensus.cc:374] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:45.529834 21991 raft_consensus.cc:729] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1d88edfc7745899396481faee4e5da, State: Initialized, Role: FOLLOWER
I20260924 15:59:45.530405 21995 tablet_bootstrap.cc:654] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:45.530565 21991 consensus_queue.cc:260] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } }
I20260924 15:59:45.531737 21995 log.cc:824] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:45.531821 21838 heartbeater.cc:503] Master 127.16.233.190:37225 was elected leader, sending a full tablet report...
I20260924 15:59:45.532399 21991 ts_tablet_manager.cc:1428] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260924 15:59:45.534039 21992 raft_consensus.cc:348] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } }
I20260924 15:59:45.534569 21992 raft_consensus.cc:374] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:45.534672 21992 raft_consensus.cc:729] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57035562f344e8885e3ddcd5763de81, State: Initialized, Role: FOLLOWER
I20260924 15:59:45.535151 21995 tablet_bootstrap.cc:492] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: No bootstrap required, opened a new log
I20260924 15:59:45.535478 21995 ts_tablet_manager.cc:1397] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 15:59:45.535331 21992 consensus_queue.cc:260] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } }
I20260924 15:59:45.537093 21992 ts_tablet_manager.cc:1428] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 15:59:45.544059 21995 raft_consensus.cc:348] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } }
I20260924 15:59:45.544545 21995 raft_consensus.cc:374] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:45.544661 21995 raft_consensus.cc:729] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82666662f11947dc9ff4aa39fdc574f6, State: Initialized, Role: FOLLOWER
I20260924 15:59:45.545252 21995 consensus_queue.cc:260] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } }
I20260924 15:59:45.546335 21973 heartbeater.cc:503] Master 127.16.233.190:37225 was elected leader, sending a full tablet report...
I20260924 15:59:45.546849 21995 ts_tablet_manager.cc:1428] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 15:59:45.606706 21998 raft_consensus.cc:482] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:45.606976 21998 raft_consensus.cc:504] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } }
I20260924 15:59:45.608686 21998 leader_election.cc:288] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c1d88edfc7745899396481faee4e5da (127.16.233.130:43665), 82666662f11947dc9ff4aa39fdc574f6 (127.16.233.131:43805)
I20260924 15:59:45.616339 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.609251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:59419 (local address 127.16.233.131:43805)
0924 15:59:45.609557 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.609565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.610345 (+ 780us) server_negotiation.cc:407] Connection header received
0924 15:59:45.610571 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.610580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.610638 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.610715 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:45.611481 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.612370 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.614500 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.615295 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.615488 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.615559 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.615928 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.616062 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260924 15:59:45.616335 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.609270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:58789 (local address 127.16.233.130:43665)
0924 15:59:45.609527 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.609537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.610361 (+ 824us) server_negotiation.cc:407] Connection header received
0924 15:59:45.610594 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.610598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.610660 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.610771 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:59:45.611529 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.612366 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.614455 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.615260 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.615413 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.615502 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.615901 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.616043 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 15:59:45.616509 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.609099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:43805 (local address 127.16.233.129:59419)
0924 15:59:45.610173 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:45.610200 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:45.610405 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:45.610756 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:45.610760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:45.610785 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:45.611321 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.611333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.612509 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.612517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.614280 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.614294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.615553 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.615560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.615735 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.615766 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 15:59:45.615874 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 15:59:45.616026 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":31,"thread_start_us":662,"threads_started":1}
I20260924 15:59:45.616544 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.609192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:43665 (local address 127.16.233.129:58789)
0924 15:59:45.610193 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:45.610217 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:45.610428 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:45.610813 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:45.610817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:45.610847 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:59:45.611330 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.611341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.612515 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.612521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.614287 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:45.614324 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.615501 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.615509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:45.615725 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.615758 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 15:59:45.615861 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 15:59:45.616028 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":107,"thread_start_us":552,"threads_started":1}
I20260924 15:59:45.617643 21927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e875e20428422897bf804dd9bb1f24" candidate_uuid: "a57035562f344e8885e3ddcd5763de81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82666662f11947dc9ff4aa39fdc574f6" is_pre_election: true
I20260924 15:59:45.617669 21792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e875e20428422897bf804dd9bb1f24" candidate_uuid: "a57035562f344e8885e3ddcd5763de81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1d88edfc7745899396481faee4e5da" is_pre_election: true
I20260924 15:59:45.618163 21927 raft_consensus.cc:2454] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57035562f344e8885e3ddcd5763de81 in term 0.
I20260924 15:59:45.618163 21792 raft_consensus.cc:2454] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57035562f344e8885e3ddcd5763de81 in term 0.
I20260924 15:59:45.618888 21591 leader_election.cc:302] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82666662f11947dc9ff4aa39fdc574f6, a57035562f344e8885e3ddcd5763de81; no voters:
I20260924 15:59:45.619350 21998 raft_consensus.cc:2781] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:45.619495 21998 raft_consensus.cc:482] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:45.619617 21998 raft_consensus.cc:3037] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:45.621856 21998 raft_consensus.cc:504] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } }
I20260924 15:59:45.622615 21998 leader_election.cc:288] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [CANDIDATE]: Term 1 election: Requested vote from peers 9c1d88edfc7745899396481faee4e5da (127.16.233.130:43665), 82666662f11947dc9ff4aa39fdc574f6 (127.16.233.131:43805)
I20260924 15:59:45.623104 21792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e875e20428422897bf804dd9bb1f24" candidate_uuid: "a57035562f344e8885e3ddcd5763de81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1d88edfc7745899396481faee4e5da"
I20260924 15:59:45.623229 21927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e875e20428422897bf804dd9bb1f24" candidate_uuid: "a57035562f344e8885e3ddcd5763de81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82666662f11947dc9ff4aa39fdc574f6"
I20260924 15:59:45.623375 21792 raft_consensus.cc:3037] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:45.623430 21927 raft_consensus.cc:3037] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:45.626791 21792 raft_consensus.cc:2454] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57035562f344e8885e3ddcd5763de81 in term 1.
I20260924 15:59:45.626788 21927 raft_consensus.cc:2454] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57035562f344e8885e3ddcd5763de81 in term 1.
I20260924 15:59:45.627327 21592 leader_election.cc:302] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 82666662f11947dc9ff4aa39fdc574f6, 9c1d88edfc7745899396481faee4e5da, a57035562f344e8885e3ddcd5763de81; no voters:
I20260924 15:59:45.627708 21998 raft_consensus.cc:2781] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:45.628163 21998 raft_consensus.cc:686] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 1 LEADER]: Becoming Leader. State: Replica: a57035562f344e8885e3ddcd5763de81, State: Running, Role: LEADER
I20260924 15:59:45.628712 21998 consensus_queue.cc:237] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } }
W20260924 15:59:45.629935 21704 tablet.cc:2369] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:59:45.633670 21513 catalog_manager.cc:5495] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a57035562f344e8885e3ddcd5763de81 (127.16.233.129). New cstate: current_term: 1 leader_uuid: "a57035562f344e8885e3ddcd5763de81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57035562f344e8885e3ddcd5763de81" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 } health_report { overall_health: UNKNOWN } } }
W20260924 15:59:45.649077 21839 tablet.cc:2369] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:59:45.652345 21974 tablet.cc:2369] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:59:45.680779 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.675822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.16.233.129:33313)
0924 15:59:45.676015 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.676020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.676037 (+ 17us) server_negotiation.cc:407] Connection header received
0924 15:59:45.676125 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.676129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.676244 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.676389 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:59:45.676733 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.677513 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.678822 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.679198 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.679291 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.679687 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:59:45.680205 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:59:45.680313 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.680404 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.680468 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 15:59:45.689429 21792 raft_consensus.cc:1261] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [term 1 FOLLOWER]: Refusing update from remote peer a57035562f344e8885e3ddcd5763de81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:45.689429 21927 raft_consensus.cc:1261] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [term 1 FOLLOWER]: Refusing update from remote peer a57035562f344e8885e3ddcd5763de81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:45.690640 22002 consensus_queue.cc:1035] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82666662f11947dc9ff4aa39fdc574f6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:45.691015 21998 consensus_queue.cc:1035] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d88edfc7745899396481faee4e5da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:45.702831 22005 mvcc.cc:205] Tried to move back new op lower bound from 7332927838972198912 to 7332927838738878464. Current Snapshot: MvccSnapshot[applied={T|T < 7332927838972198912}]
I20260924 15:59:45.705838 22006 mvcc.cc:205] Tried to move back new op lower bound from 7332927838972198912 to 7332927838738878464. Current Snapshot: MvccSnapshot[applied={T|T < 7332927838972198912 or (T in {7332927838972198912})}]
I20260924 15:59:45.718259 21511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:45.718636 21511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:45.722503 21511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8b50d9712784862bec56997d3dca1b4: Sending DeleteTablet for 3 replicas of tablet d5e875e20428422897bf804dd9bb1f24
I20260924 15:59:45.723767 21772 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e875e20428422897bf804dd9bb1f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:45 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:38498
I20260924 15:59:45.723888 21907 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e875e20428422897bf804dd9bb1f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:45 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:58414
I20260924 15:59:45.724880 21638 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e875e20428422897bf804dd9bb1f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:45 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:35228
I20260924 15:59:45.725427 22015 tablet_replica.cc:330] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: stopping tablet replica
I20260924 15:59:45.725797 22017 tablet_replica.cc:330] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: stopping tablet replica
I20260924 15:59:45.725811 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 21574
I20260924 15:59:45.725956 22015 raft_consensus.cc:2229] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:45.726209 22017 raft_consensus.cc:2229] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:59:45.726577 22015 raft_consensus.cc:2258] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:45.727138 22017 raft_consensus.cc:2258] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:45.730173 22017 ts_tablet_manager.cc:1905] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:45.736337 22015 ts_tablet_manager.cc:1905] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:45.737490 22016 tablet_replica.cc:330] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: stopping tablet replica
I20260924 15:59:45.737918 22016 raft_consensus.cc:2229] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:45.738178 22017 ts_tablet_manager.cc:1918] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:59:45.738441 22017 log.cc:1196] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/d5e875e20428422897bf804dd9bb1f24
I20260924 15:59:45.738508 22016 raft_consensus.cc:2258] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:45.739159 22017 ts_tablet_manager.cc:1939] T d5e875e20428422897bf804dd9bb1f24 P a57035562f344e8885e3ddcd5763de81: Deleting consensus metadata
I20260924 15:59:45.741120 21498 catalog_manager.cc:4841] TS a57035562f344e8885e3ddcd5763de81 (127.16.233.129:33313): tablet d5e875e20428422897bf804dd9bb1f24 (table test-table [id=7712e2b1bf1f4d11b77b9ca6aca35082]) successfully deleted
I20260924 15:59:45.742691 22016 ts_tablet_manager.cc:1905] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:45.744217 22015 ts_tablet_manager.cc:1918] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:59:45.744391 22015 log.cc:1196] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/d5e875e20428422897bf804dd9bb1f24
I20260924 15:59:45.745062 22015 ts_tablet_manager.cc:1939] T d5e875e20428422897bf804dd9bb1f24 P 9c1d88edfc7745899396481faee4e5da: Deleting consensus metadata
I20260924 15:59:45.749471 22016 ts_tablet_manager.cc:1918] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:59:45.749630 22016 log.cc:1196] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/d5e875e20428422897bf804dd9bb1f24
I20260924 15:59:45.750120 22016 ts_tablet_manager.cc:1939] T d5e875e20428422897bf804dd9bb1f24 P 82666662f11947dc9ff4aa39fdc574f6: Deleting consensus metadata
I20260924 15:59:45.752241 21498 catalog_manager.cc:4841] TS 82666662f11947dc9ff4aa39fdc574f6 (127.16.233.131:43805): tablet d5e875e20428422897bf804dd9bb1f24 (table test-table [id=7712e2b1bf1f4d11b77b9ca6aca35082]) successfully deleted
I20260924 15:59:45.753909 21499 catalog_manager.cc:4841] TS 9c1d88edfc7745899396481faee4e5da (127.16.233.130:43665): tablet d5e875e20428422897bf804dd9bb1f24 (table test-table [id=7712e2b1bf1f4d11b77b9ca6aca35082]) successfully deleted
Sep 24 15:59:45 dist-test-slave-xn4h krb5kdc[21467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:45.762887 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.728188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.16.233.129:33313)
0924 15:59:45.728353 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.728359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.742980 (+ 14621us) server_negotiation.cc:407] Connection header received
0924 15:59:45.743327 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.743334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.743418 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.743491 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:45.745888 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.747141 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.749085 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.749757 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.749892 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.755420 (+ 5528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:45.755442 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:45.755462 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:45.755527 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:45.759584 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:45.760489 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:45.760499 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:45.760504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:45.760677 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:45.761113 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:45.761119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:45.761123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:45.761570 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:45.761876 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.762205 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.762491 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 15:59:45.763787 21698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:45.864158 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 21574
I20260924 15:59:45.883163 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 21709
Sep 24 15:59:45 dist-test-slave-xn4h krb5kdc[21467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 15:59:45.898658 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:45.884064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.16.233.130:43665)
0924 15:59:45.884273 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:45.884278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:45.884300 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:59:45.884438 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:45.884446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:45.884540 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:45.884653 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:45.885973 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.886822 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.887994 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:45.888369 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:45.888460 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:45.892651 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:45.892666 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:45.892682 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:45.892718 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:45.895749 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:45.896622 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:45.896645 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:45.896647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:45.896778 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:45.897171 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:45.897174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:45.897176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:45.897552 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:45.897781 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:45.898119 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 15:59:45.898296 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 15:59:45.899467 21833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:45.989010 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 21709
I20260924 15:59:46.005697 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 21844
Sep 24 15:59:46 dist-test-slave-xn4h krb5kdc[21467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 15:59:46.020083 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:46.006475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.16.233.131:43805)
0924 15:59:46.006657 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:46.006662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:46.006680 (+ 18us) server_negotiation.cc:407] Connection header received
0924 15:59:46.006823 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:46.006827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:46.006891 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:46.006974 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:46.008173 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:46.008881 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:46.010068 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:46.010613 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:46.010707 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:46.015093 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:46.015109 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:46.015126 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:46.015164 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:46.017685 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:46.018259 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:46.018266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:46.018268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:46.018450 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:46.018771 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:46.018774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:46.018776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:46.019104 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:46.019320 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:46.019568 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 15:59:46.019730 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 15:59:46.020815 21968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:46.110396 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 21844
I20260924 15:59:46.129315 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 21480
I20260924 15:59:46.130546 21543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:46.239239 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 21480
2026-09-24T15:59:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4204 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T15:59:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:46Z Disabled control of system clock
I20260924 15:59:46.307215 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36957
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:59:46.636718 22026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:46.637014 22026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:46.637099 22026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:46.644310 22026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:46.644441 22026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:46.644480 22026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:46.644537 22026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:46.644583 22026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:46.654021 22026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:46.655479 22026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:46.657239 22026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:46.669240 22032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:46.669348 22031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:46.669296 22034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:46.670070 22026 server_base.cc:1034] running on GCE node
I20260924 15:59:46.670938 22026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:46.672363 22026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:46.673657 22026 hybrid_clock.cc:648] HybridClock initialized: now 1790265586673543 us; error 103 us; skew 500 ppm
I20260924 15:59:46.674073 22026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 15:59:46.676556 22026 webserver.cc:460] Webserver started at http://127.16.233.190:39889/ using document root <none> and password file <none>
I20260924 15:59:46.677306 22026 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:46.677433 22026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:46.677788 22026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:46.680884 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "f2124730eb014960abeb44793d09d87f"
format_stamp: "Formatted at 2026-09-24 15:59:46 on dist-test-slave-xn4h"
server_key: "d060de3d0511c6bf3a9282f4b23bbc04"
server_key_iv: "c38c0fb663d6888cf943070090e9cfc8"
server_key_version: "encryptionkey@0"
I20260924 15:59:46.681721 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "f2124730eb014960abeb44793d09d87f"
format_stamp: "Formatted at 2026-09-24 15:59:46 on dist-test-slave-xn4h"
server_key: "d060de3d0511c6bf3a9282f4b23bbc04"
server_key_iv: "c38c0fb663d6888cf943070090e9cfc8"
server_key_version: "encryptionkey@0"
I20260924 15:59:46.687345 22026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 15:59:46.691011 22040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:46.692447 22026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:46.692632 22026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "f2124730eb014960abeb44793d09d87f"
format_stamp: "Formatted at 2026-09-24 15:59:46 on dist-test-slave-xn4h"
server_key: "d060de3d0511c6bf3a9282f4b23bbc04"
server_key_iv: "c38c0fb663d6888cf943070090e9cfc8"
server_key_version: "encryptionkey@0"
I20260924 15:59:46.692871 22026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:46.698918 22026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:46.699708 22026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:46.700014 22026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:46.718472 22026 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:36957
I20260924 15:59:46.718490 22092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:36957 every 8 connection(s)
I20260924 15:59:46.719910 22026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:46.723776 22093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:46.726598 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 22026
I20260924 15:59:46.726826 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:46.727634 17318 external_mini_cluster.cc:1428] Setting key fa4af4172f3bec9510b8a8de9811962e
I20260924 15:59:46.736119 22093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f: Bootstrap starting.
I20260924 15:59:46.739166 22093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:46.740157 22093 log.cc:824] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:46.743316 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:46.729813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.16.233.190:36957)
0924 15:59:46.730846 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:46.730857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:46.730896 (+ 39us) server_negotiation.cc:407] Connection header received
0924 15:59:46.732471 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:46.732493 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:46.732930 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:46.733533 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:46.741366 (+ 7833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:46.741375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:46.741392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:46.741416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:46.741502 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:46.741529 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:46.741719 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:46.741842 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 15:59:46.742108 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":375,"threads_started":1}
I20260924 15:59:46.743425 22093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f: No bootstrap required, opened a new log
I20260924 15:59:46.751272 22093 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2124730eb014960abeb44793d09d87f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36957 } }
I20260924 15:59:46.751587 22093 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:46.751657 22093 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2124730eb014960abeb44793d09d87f, State: Initialized, Role: FOLLOWER
I20260924 15:59:46.752138 22093 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2124730eb014960abeb44793d09d87f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36957 } }
I20260924 15:59:46.752323 22093 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:46.752413 22093 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:46.752528 22093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:46.754738 22093 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2124730eb014960abeb44793d09d87f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36957 } }
I20260924 15:59:46.755264 22093 leader_election.cc:302] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2124730eb014960abeb44793d09d87f; no voters:
I20260924 15:59:46.755861 22093 leader_election.cc:288] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:46.756022 22098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:46.756747 22098 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [term 1 LEADER]: Becoming Leader. State: Replica: f2124730eb014960abeb44793d09d87f, State: Running, Role: LEADER
I20260924 15:59:46.757292 22098 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2124730eb014960abeb44793d09d87f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36957 } }
I20260924 15:59:46.757977 22093 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:46.760269 22099 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2124730eb014960abeb44793d09d87f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2124730eb014960abeb44793d09d87f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36957 } } }
I20260924 15:59:46.760387 22100 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2124730eb014960abeb44793d09d87f. Latest consensus state: current_term: 1 leader_uuid: "f2124730eb014960abeb44793d09d87f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2124730eb014960abeb44793d09d87f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36957 } } }
I20260924 15:59:46.760699 22099 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:46.760772 22100 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:46.763100 22103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:46.768811 22103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:46.777442 22103 catalog_manager.cc:1347] Generated new cluster ID: d22ad60984384b409ff075e7b79f6036
I20260924 15:59:46.777568 22103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:46.801503 22103 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:46.802060 22103 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:46.819634 22103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f: Generated new TSK 0
I20260924 15:59:46.820590 22103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:59:46.832928 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36957
--builtin_ntp_servers=127.16.233.148:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:59:47.183066 22117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:47.183346 22117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:47.183537 22117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:47.191231 22117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:47.191334 22117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:47.191498 22117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:47.201172 22117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:47.202656 22117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:47.204712 22117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:47.217911 22122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:47.217931 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:47.217875 22123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:47.218681 22117 server_base.cc:1034] running on GCE node
I20260924 15:59:47.219310 22117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:47.220273 22117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:47.221498 22117 hybrid_clock.cc:648] HybridClock initialized: now 1790265587221437 us; error 49 us; skew 500 ppm
I20260924 15:59:47.221820 22117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 15:59:47.224220 22117 webserver.cc:460] Webserver started at http://127.16.233.129:45211/ using document root <none> and password file <none>
I20260924 15:59:47.224920 22117 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:47.225052 22117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:47.225390 22117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:47.228225 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "ce98631236504b618be45ba32e777b40"
format_stamp: "Formatted at 2026-09-24 15:59:47 on dist-test-slave-xn4h"
server_key: "580f12d1ecce93375d49dd84f7ab0fc9"
server_key_iv: "8c37bb264ddcf576301810ee48c04b74"
server_key_version: "encryptionkey@0"
I20260924 15:59:47.229275 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "ce98631236504b618be45ba32e777b40"
format_stamp: "Formatted at 2026-09-24 15:59:47 on dist-test-slave-xn4h"
server_key: "580f12d1ecce93375d49dd84f7ab0fc9"
server_key_iv: "8c37bb264ddcf576301810ee48c04b74"
server_key_version: "encryptionkey@0"
I20260924 15:59:47.234524 22117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 15:59:47.237866 22131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:47.239257 22117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:47.239447 22117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "ce98631236504b618be45ba32e777b40"
format_stamp: "Formatted at 2026-09-24 15:59:47 on dist-test-slave-xn4h"
server_key: "580f12d1ecce93375d49dd84f7ab0fc9"
server_key_iv: "8c37bb264ddcf576301810ee48c04b74"
server_key_version: "encryptionkey@0"
I20260924 15:59:47.239674 22117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:47.245378 22117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:47.246160 22117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:47.246462 22117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:47.247347 22117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:47.248683 22117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:47.248816 22117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:47.248939 22117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:47.249025 22117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:47.281939 22117 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:33651
I20260924 15:59:47.281996 22245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:33651 every 8 connection(s)
I20260924 15:59:47.283334 22117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:47.285570 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 22117
I20260924 15:59:47.285748 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:47.286602 17318 external_mini_cluster.cc:1428] Setting key 722538fbc6e4b91d7763f7aedd8125e3
I20260924 15:59:47.289239 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36957
--builtin_ntp_servers=127.16.233.148:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 15:59:47.292016 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:47.285957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:34281 (local address 127.16.233.190:36957)
0924 15:59:47.286479 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:47.286487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:47.287932 (+ 1445us) server_negotiation.cc:407] Connection header received
0924 15:59:47.289644 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:47.289649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:47.289716 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:47.289842 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:47.291247 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:47.291252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:47.291257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:47.291276 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:47.291325 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:47.291337 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:47.291467 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:47.291661 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 15:59:47.291731 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":225,"threads_started":1}
I20260924 15:59:47.292949 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:47.286645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36957 (local address 127.16.233.129:34281)
0924 15:59:47.287763 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:47.287809 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:47.289317 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:47.290080 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:47.290088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:47.290179 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:47.290938 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:59:47.290941 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:47.290984 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:59:47.291067 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:47.291439 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:47.291450 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:47.291458 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 15:59:47.291750 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 15:59:47.292008 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":335,"threads_started":1}
I20260924 15:59:47.294499 22246 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36957
I20260924 15:59:47.294850 22246 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:47.295511 22246 heartbeater.cc:511] Master 127.16.233.190:36957 requested a full tablet report, sending...
I20260924 15:59:47.297607 22058 ts_manager.cc:194] Registered new tserver with Master: ce98631236504b618be45ba32e777b40 (127.16.233.129:33651)
W20260924 15:59:47.611770 22250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:47.612112 22250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:47.612288 22250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:47.619777 22250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:47.619937 22250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:47.620146 22250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:47.629334 22250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:47.630741 22250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:47.632611 22250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:47.646073 22256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:47.646054 22258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:47.646121 22255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:47.646822 22250 server_base.cc:1034] running on GCE node
I20260924 15:59:47.647454 22250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:47.648417 22250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:47.649652 22250 hybrid_clock.cc:648] HybridClock initialized: now 1790265587649597 us; error 46 us; skew 500 ppm
I20260924 15:59:47.649982 22250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 15:59:47.652263 22250 webserver.cc:460] Webserver started at http://127.16.233.130:35565/ using document root <none> and password file <none>
I20260924 15:59:47.652992 22250 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:47.653122 22250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:47.653448 22250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:47.656315 22250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "2b3e38ea389f440ba244f80a79d6a616"
format_stamp: "Formatted at 2026-09-24 15:59:47 on dist-test-slave-xn4h"
server_key: "16d63997728329c46bc8072de49f6ed0"
server_key_iv: "4df6f1d9dba4531af6ee7abe5178b627"
server_key_version: "encryptionkey@0"
I20260924 15:59:47.657127 22250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "2b3e38ea389f440ba244f80a79d6a616"
format_stamp: "Formatted at 2026-09-24 15:59:47 on dist-test-slave-xn4h"
server_key: "16d63997728329c46bc8072de49f6ed0"
server_key_iv: "4df6f1d9dba4531af6ee7abe5178b627"
server_key_version: "encryptionkey@0"
I20260924 15:59:47.662457 22250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 15:59:47.665853 22264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:47.667264 22250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:47.667452 22250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "2b3e38ea389f440ba244f80a79d6a616"
format_stamp: "Formatted at 2026-09-24 15:59:47 on dist-test-slave-xn4h"
server_key: "16d63997728329c46bc8072de49f6ed0"
server_key_iv: "4df6f1d9dba4531af6ee7abe5178b627"
server_key_version: "encryptionkey@0"
I20260924 15:59:47.667692 22250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:47.673516 22250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:47.674389 22250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:47.674664 22250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:47.675575 22250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:47.676990 22250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:47.677112 22250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:47.677250 22250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:47.677345 22250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:47.710579 22250 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:41849
I20260924 15:59:47.710630 22378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:41849 every 8 connection(s)
I20260924 15:59:47.711951 22250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:47.719731 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 22250
I20260924 15:59:47.719915 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:47.720505 17318 external_mini_cluster.cc:1428] Setting key 3cfc13bd58a903ee41e22d07ceb544fa
I20260924 15:59:47.720566 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:47.714505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:59457 (local address 127.16.233.190:36957)
0924 15:59:47.714689 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:47.714702 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:47.716137 (+ 1435us) server_negotiation.cc:407] Connection header received
0924 15:59:47.717921 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:47.717927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:47.718018 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:47.718147 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:47.719683 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:47.719689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:47.719694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:47.719715 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:47.719763 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:47.719779 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:47.719906 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:47.720207 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 15:59:47.720279 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 15:59:47.721602 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:47.715052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36957 (local address 127.16.233.130:59457)
0924 15:59:47.715970 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:47.716006 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:47.717635 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:47.718462 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:47.718471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:47.718562 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:47.719306 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:59:47.719310 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:47.719360 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:59:47.719476 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:47.719958 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:47.719969 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:47.719977 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 15:59:47.720244 (+ 267us) client_negotiation.cc:240] Negotiation successful
0924 15:59:47.720491 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":267,"threads_started":1}
I20260924 15:59:47.723227 22379 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36957
I20260924 15:59:47.723555 22379 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:47.723757 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36957
--builtin_ntp_servers=127.16.233.148:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 15:59:47.724220 22379 heartbeater.cc:511] Master 127.16.233.190:36957 requested a full tablet report, sending...
I20260924 15:59:47.725827 22058 ts_manager.cc:194] Registered new tserver with Master: 2b3e38ea389f440ba244f80a79d6a616 (127.16.233.130:41849)
W20260924 15:59:48.049230 22382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:48.049510 22382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:48.049698 22382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:48.057019 22382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:48.057119 22382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:48.057276 22382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:48.066684 22382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:48.068034 22382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:48.070099 22382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:48.086731 22387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:48.086676 22390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:48.086643 22388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:48.087555 22382 server_base.cc:1034] running on GCE node
I20260924 15:59:48.088202 22382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:48.089207 22382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:48.090449 22382 hybrid_clock.cc:648] HybridClock initialized: now 1790265588090396 us; error 54 us; skew 500 ppm
I20260924 15:59:48.090782 22382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 15:59:48.093044 22382 webserver.cc:460] Webserver started at http://127.16.233.131:40423/ using document root <none> and password file <none>
I20260924 15:59:48.093755 22382 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:48.093883 22382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:48.094224 22382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:48.097254 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "c1255a571ef84d21b42d0df00fe6dbe6"
format_stamp: "Formatted at 2026-09-24 15:59:48 on dist-test-slave-xn4h"
server_key: "f927aea52a9ef74c978111caeebfd392"
server_key_iv: "6d2caa406559f9ce968b7def98cc5f1b"
server_key_version: "encryptionkey@0"
I20260924 15:59:48.098084 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "c1255a571ef84d21b42d0df00fe6dbe6"
format_stamp: "Formatted at 2026-09-24 15:59:48 on dist-test-slave-xn4h"
server_key: "f927aea52a9ef74c978111caeebfd392"
server_key_iv: "6d2caa406559f9ce968b7def98cc5f1b"
server_key_version: "encryptionkey@0"
I20260924 15:59:48.103260 22382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 15:59:48.106736 22396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:48.108122 22382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 15:59:48.108327 22382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "c1255a571ef84d21b42d0df00fe6dbe6"
format_stamp: "Formatted at 2026-09-24 15:59:48 on dist-test-slave-xn4h"
server_key: "f927aea52a9ef74c978111caeebfd392"
server_key_iv: "6d2caa406559f9ce968b7def98cc5f1b"
server_key_version: "encryptionkey@0"
I20260924 15:59:48.108569 22382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:48.114396 22382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:48.115229 22382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:48.115512 22382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:48.116437 22382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:48.118115 22382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:48.118250 22382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:48.118381 22382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:48.118460 22382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:48.155243 22382 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:33019
I20260924 15:59:48.155293 22509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:33019 every 8 connection(s)
I20260924 15:59:48.156644 22382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:48.164005 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 22382
I20260924 15:59:48.164197 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:48.164727 17318 external_mini_cluster.cc:1428] Setting key d30d848f00b4dd66bdab3be0c495f9b8
I20260924 15:59:48.165421 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.159489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:41467 (local address 127.16.233.190:36957)
0924 15:59:48.159711 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.159718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.161216 (+ 1498us) server_negotiation.cc:407] Connection header received
0924 15:59:48.162912 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.162917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.162981 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.163071 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.164600 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.164607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.164611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.164632 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.164672 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.164689 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.164839 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.165056 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.165128 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 15:59:48.166675 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.160104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36957 (local address 127.16.233.131:41467)
0924 15:59:48.161050 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.161085 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:48.162673 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:48.163285 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:48.163293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:48.163394 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:48.164210 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:59:48.164213 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:48.164289 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:59:48.164396 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:48.164824 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:48.164838 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:48.164846 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 15:59:48.165126 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 15:59:48.165400 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":300,"threads_started":1}
I20260924 15:59:48.168102 22510 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36957
I20260924 15:59:48.168414 22510 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:48.168979 22510 heartbeater.cc:511] Master 127.16.233.190:36957 requested a full tablet report, sending...
I20260924 15:59:48.170264 22058 ts_manager.cc:194] Registered new tserver with Master: c1255a571ef84d21b42d0df00fe6dbe6 (127.16.233.131:33019)
I20260924 15:59:48.180505 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:59:48.187402 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.185035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.16.233.190:36957)
0924 15:59:48.185284 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.185293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.185638 (+ 345us) server_negotiation.cc:407] Connection header received
0924 15:59:48.185768 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.185773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.185828 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.185912 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.186817 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.186821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.186824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.186838 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.186869 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.186880 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.186982 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.187037 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.187081 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 15:59:48.191179 22058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:59:48.197904 22058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:59:48.216846 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.209806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:33651 (local address 127.0.0.1:37882)
0924 15:59:48.210720 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.210790 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:48.210972 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:48.212903 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:48.212914 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:48.213040 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:48.214051 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:59:48.214057 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:48.214096 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:59:48.214206 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:48.215874 (+ 1668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:48.215890 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:48.215904 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:59:48.216114 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 15:59:48.216225 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":313,"threads_started":1}
I20260924 15:59:48.218791 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.210028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.16.233.129:33651)
0924 15:59:48.212107 (+ 2079us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.212119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.212217 (+ 98us) server_negotiation.cc:407] Connection header received
0924 15:59:48.212311 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.212322 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.212645 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.213811 (+ 1166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.215542 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.215556 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.215575 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.215606 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.215669 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.215695 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.218062 (+ 2367us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.218165 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.218299 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1897,"thread_start_us":1775,"threads_started":1}
I20260924 15:59:48.224313 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.210981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33019 (local address 127.0.0.1:35624)
0924 15:59:48.217133 (+ 6152us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.217155 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:48.217250 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:48.221780 (+ 4530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:48.221784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:48.221828 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:48.222493 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:59:48.222498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:48.222518 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:59:48.222566 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:48.223867 (+ 1301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:48.223874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:48.223881 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:59:48.223976 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 15:59:48.224029 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6027,"thread_start_us":1224,"threads_started":1}
I20260924 15:59:48.224128 22180 tablet_service.cc:1467] Processing CreateTablet for tablet cff7260ffc2b408ab3e6527061c0b9d9 (DEFAULT_TABLE table=test-table [id=9f14b649d505406dabdbbbfdf1d56a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:48.225072 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.210683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41849 (local address 127.0.0.1:41512)
0924 15:59:48.211494 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.211517 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:48.211615 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:48.223269 (+ 11654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:48.223274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:48.223297 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:48.223951 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:59:48.223954 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:48.223969 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:59:48.224010 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:48.224452 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:48.224457 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:48.224462 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:59:48.224634 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 15:59:48.224687 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":10626,"threads_started":1}
I20260924 15:59:48.225215 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.211723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.16.233.131:33019)
0924 15:59:48.221184 (+ 9461us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.221203 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.221245 (+ 42us) server_negotiation.cc:407] Connection header received
0924 15:59:48.221326 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.221335 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.221585 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.222588 (+ 1003us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.223579 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.223590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.223607 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.223627 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.223685 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.223727 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.224651 (+ 924us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.224729 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.224831 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9274,"thread_start_us":9094,"threads_started":1}
I20260924 15:59:48.225616 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.211362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.16.233.130:41849)
0924 15:59:48.222757 (+ 11395us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.222772 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.222802 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:59:48.222873 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.222881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.223066 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.223234 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.224178 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.224187 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.224212 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.224238 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.224296 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.224321 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.224464 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.224775 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.224856 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11231,"thread_start_us":12603,"threads_started":1}
I20260924 15:59:48.226256 22180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff7260ffc2b408ab3e6527061c0b9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:48.228549 22445 tablet_service.cc:1467] Processing CreateTablet for tablet cff7260ffc2b408ab3e6527061c0b9d9 (DEFAULT_TABLE table=test-table [id=9f14b649d505406dabdbbbfdf1d56a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:48.229710 22445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff7260ffc2b408ab3e6527061c0b9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:48.229359 22313 tablet_service.cc:1467] Processing CreateTablet for tablet cff7260ffc2b408ab3e6527061c0b9d9 (DEFAULT_TABLE table=test-table [id=9f14b649d505406dabdbbbfdf1d56a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:59:48.230894 22313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff7260ffc2b408ab3e6527061c0b9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:48.240549 22528 tablet_bootstrap.cc:492] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: Bootstrap starting.
I20260924 15:59:48.242931 22529 tablet_bootstrap.cc:492] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40: Bootstrap starting.
I20260924 15:59:48.244836 22528 tablet_bootstrap.cc:654] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:48.246121 22530 tablet_bootstrap.cc:492] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: Bootstrap starting.
I20260924 15:59:48.246173 22528 log.cc:824] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:48.246484 22529 tablet_bootstrap.cc:654] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:48.247583 22529 log.cc:824] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:48.249445 22530 tablet_bootstrap.cc:654] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:48.249610 22528 tablet_bootstrap.cc:492] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: No bootstrap required, opened a new log
I20260924 15:59:48.249902 22528 ts_tablet_manager.cc:1397] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 15:59:48.250514 22529 tablet_bootstrap.cc:492] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40: No bootstrap required, opened a new log
I20260924 15:59:48.250545 22530 log.cc:824] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:48.250761 22529 ts_tablet_manager.cc:1397] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260924 15:59:48.253427 22530 tablet_bootstrap.cc:492] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: No bootstrap required, opened a new log
I20260924 15:59:48.253683 22530 ts_tablet_manager.cc:1397] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260924 15:59:48.260527 22530 raft_consensus.cc:348] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } }
I20260924 15:59:48.260581 22528 raft_consensus.cc:348] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } }
I20260924 15:59:48.260924 22530 raft_consensus.cc:374] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:48.261009 22530 raft_consensus.cc:729] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3e38ea389f440ba244f80a79d6a616, State: Initialized, Role: FOLLOWER
I20260924 15:59:48.261036 22528 raft_consensus.cc:374] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:48.261140 22528 raft_consensus.cc:729] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1255a571ef84d21b42d0df00fe6dbe6, State: Initialized, Role: FOLLOWER
I20260924 15:59:48.261107 22529 raft_consensus.cc:348] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } }
I20260924 15:59:48.261524 22529 raft_consensus.cc:374] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:48.261620 22529 raft_consensus.cc:729] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce98631236504b618be45ba32e777b40, State: Initialized, Role: FOLLOWER
I20260924 15:59:48.261786 22528 consensus_queue.cc:260] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } }
I20260924 15:59:48.261806 22530 consensus_queue.cc:260] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } }
I20260924 15:59:48.262147 22529 consensus_queue.cc:260] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } }
I20260924 15:59:48.263269 22510 heartbeater.cc:503] Master 127.16.233.190:36957 was elected leader, sending a full tablet report...
I20260924 15:59:48.264057 22528 ts_tablet_manager.cc:1428] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260924 15:59:48.265717 22379 heartbeater.cc:503] Master 127.16.233.190:36957 was elected leader, sending a full tablet report...
I20260924 15:59:48.266338 22246 heartbeater.cc:503] Master 127.16.233.190:36957 was elected leader, sending a full tablet report...
I20260924 15:59:48.266964 22529 ts_tablet_manager.cc:1428] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260924 15:59:48.268101 22530 ts_tablet_manager.cc:1428] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: Time spent starting tablet: real 0.014s user 0.004s sys 0.008s
W20260924 15:59:48.285563 22247 tablet.cc:2369] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:59:48.307336 22536 raft_consensus.cc:482] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:59:48.307690 22536 raft_consensus.cc:504] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } }
I20260924 15:59:48.309398 22536 leader_election.cc:288] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1255a571ef84d21b42d0df00fe6dbe6 (127.16.233.131:33019), 2b3e38ea389f440ba244f80a79d6a616 (127.16.233.130:41849)
I20260924 15:59:48.313058 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.309875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:60253 (local address 127.16.233.131:33019)
0924 15:59:48.310055 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.310061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.310960 (+ 899us) server_negotiation.cc:407] Connection header received
0924 15:59:48.311030 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.311034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.311087 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.311196 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.312473 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.312479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.312483 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.312504 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.312539 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.312554 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.312698 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.312752 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.312813 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 15:59:48.313155 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.309874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:40491 (local address 127.16.233.130:41849)
0924 15:59:48.310054 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.310061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.310834 (+ 773us) server_negotiation.cc:407] Connection header received
0924 15:59:48.311016 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.311022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.311087 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.311196 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.312472 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.312479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.312483 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.312503 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.312539 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.312554 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.312674 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.312845 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.312906 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 15:59:48.313427 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.309850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41849 (local address 127.16.233.129:40491)
0924 15:59:48.310677 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.310714 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:48.310865 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:48.311246 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:48.311251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:48.311276 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:48.312249 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:59:48.312253 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:48.312263 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:59:48.312312 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:48.312658 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:48.312663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:48.312668 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:59:48.312855 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 15:59:48.312886 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":323,"threads_started":1}
I20260924 15:59:48.313601 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.309868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33019 (local address 127.16.233.129:60253)
0924 15:59:48.310833 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.310855 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:48.310945 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:48.311246 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:48.311251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:48.311276 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:48.312240 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:59:48.312244 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:48.312263 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:59:48.312312 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:48.312658 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:48.312663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:48.312668 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:59:48.312803 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 15:59:48.312866 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":50,"thread_start_us":611,"threads_started":1}
I20260924 15:59:48.314633 22465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff7260ffc2b408ab3e6527061c0b9d9" candidate_uuid: "ce98631236504b618be45ba32e777b40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" is_pre_election: true
I20260924 15:59:48.314580 22333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff7260ffc2b408ab3e6527061c0b9d9" candidate_uuid: "ce98631236504b618be45ba32e777b40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3e38ea389f440ba244f80a79d6a616" is_pre_election: true
I20260924 15:59:48.315171 22465 raft_consensus.cc:2454] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce98631236504b618be45ba32e777b40 in term 0.
I20260924 15:59:48.315177 22333 raft_consensus.cc:2454] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce98631236504b618be45ba32e777b40 in term 0.
I20260924 15:59:48.315876 22133 leader_election.cc:302] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3e38ea389f440ba244f80a79d6a616, ce98631236504b618be45ba32e777b40; no voters:
I20260924 15:59:48.316286 22536 raft_consensus.cc:2781] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:59:48.316450 22536 raft_consensus.cc:482] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:59:48.316560 22536 raft_consensus.cc:3037] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:48.318727 22536 raft_consensus.cc:504] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } }
I20260924 15:59:48.319294 22536 leader_election.cc:288] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [CANDIDATE]: Term 1 election: Requested vote from peers c1255a571ef84d21b42d0df00fe6dbe6 (127.16.233.131:33019), 2b3e38ea389f440ba244f80a79d6a616 (127.16.233.130:41849)
I20260924 15:59:48.319758 22333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff7260ffc2b408ab3e6527061c0b9d9" candidate_uuid: "ce98631236504b618be45ba32e777b40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3e38ea389f440ba244f80a79d6a616"
I20260924 15:59:48.319775 22465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff7260ffc2b408ab3e6527061c0b9d9" candidate_uuid: "ce98631236504b618be45ba32e777b40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1255a571ef84d21b42d0df00fe6dbe6"
I20260924 15:59:48.319962 22333 raft_consensus.cc:3037] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:48.319973 22465 raft_consensus.cc:3037] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:48.322173 22333 raft_consensus.cc:2454] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce98631236504b618be45ba32e777b40 in term 1.
I20260924 15:59:48.322249 22465 raft_consensus.cc:2454] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce98631236504b618be45ba32e777b40 in term 1.
I20260924 15:59:48.322688 22133 leader_election.cc:302] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3e38ea389f440ba244f80a79d6a616, ce98631236504b618be45ba32e777b40; no voters:
I20260924 15:59:48.322999 22536 raft_consensus.cc:2781] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:48.323454 22536 raft_consensus.cc:686] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 1 LEADER]: Becoming Leader. State: Replica: ce98631236504b618be45ba32e777b40, State: Running, Role: LEADER
I20260924 15:59:48.324036 22536 consensus_queue.cc:237] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } }
I20260924 15:59:48.328529 22058 catalog_manager.cc:5495] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce98631236504b618be45ba32e777b40 (127.16.233.129). New cstate: current_term: 1 leader_uuid: "ce98631236504b618be45ba32e777b40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce98631236504b618be45ba32e777b40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 33651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 } health_report { overall_health: UNKNOWN } } }
I20260924 15:59:48.348970 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.346700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.16.233.129:33651)
0924 15:59:48.346906 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.346913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.346996 (+ 83us) server_negotiation.cc:407] Connection header received
0924 15:59:48.347261 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.347266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.347370 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.347488 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.348346 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.348354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.348362 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.348378 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.348410 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.348422 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.348542 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.348629 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.348702 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 15:59:48.357326 22465 raft_consensus.cc:1261] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [term 1 FOLLOWER]: Refusing update from remote peer ce98631236504b618be45ba32e777b40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:48.357322 22333 raft_consensus.cc:1261] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [term 1 FOLLOWER]: Refusing update from remote peer ce98631236504b618be45ba32e777b40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:59:48.358333 22539 consensus_queue.cc:1035] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1255a571ef84d21b42d0df00fe6dbe6" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:48.358680 22536 consensus_queue.cc:1035] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3e38ea389f440ba244f80a79d6a616" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:59:48.373355 22542 mvcc.cc:205] Tried to move back new op lower bound from 7332927849901711360 to 7332927849779019776. Current Snapshot: MvccSnapshot[applied={T|T < 7332927849901711360}]
I20260924 15:59:48.386235 22056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:48.386591 22056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:59:48.390532 22056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2124730eb014960abeb44793d09d87f: Sending DeleteTablet for 3 replicas of tablet cff7260ffc2b408ab3e6527061c0b9d9
I20260924 15:59:48.391883 22445 tablet_service.cc:1514] Processing DeleteTablet for tablet cff7260ffc2b408ab3e6527061c0b9d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:48 UTC) from {username='slave'} at 127.0.0.1:35624
I20260924 15:59:48.391888 22180 tablet_service.cc:1514] Processing DeleteTablet for tablet cff7260ffc2b408ab3e6527061c0b9d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:48 UTC) from {username='slave'} at 127.0.0.1:37882
I20260924 15:59:48.392535 22313 tablet_service.cc:1514] Processing DeleteTablet for tablet cff7260ffc2b408ab3e6527061c0b9d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:59:48 UTC) from {username='slave'} at 127.0.0.1:41512
I20260924 15:59:48.394950 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 22117
I20260924 15:59:48.404313 22554 tablet_replica.cc:330] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: stopping tablet replica
I20260924 15:59:48.404949 22554 raft_consensus.cc:2229] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:48.405648 22554 raft_consensus.cc:2258] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:48.408787 22554 ts_tablet_manager.cc:1905] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:48.413309 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.396265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.16.233.129:33651)
0924 15:59:48.396474 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.396483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.410186 (+ 13703us) server_negotiation.cc:407] Connection header received
0924 15:59:48.410778 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.410786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.410861 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.410950 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.412388 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.412395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.412400 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.412428 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.412467 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.412483 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.412592 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.412876 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.412944 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 15:59:48.414718 22552 tablet_replica.cc:330] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40: stopping tablet replica
I20260924 15:59:48.414793 22553 tablet_replica.cc:330] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: stopping tablet replica
I20260924 15:59:48.414757 22241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:48.415306 22553 raft_consensus.cc:2229] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:59:48.415874 22553 raft_consensus.cc:2258] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:48.418227 22553 ts_tablet_manager.cc:1905] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:59:48.420241 22554 ts_tablet_manager.cc:1918] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:59:48.420439 22554 log.cc:1196] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/cff7260ffc2b408ab3e6527061c0b9d9
I20260924 15:59:48.420950 22554 ts_tablet_manager.cc:1939] T cff7260ffc2b408ab3e6527061c0b9d9 P 2b3e38ea389f440ba244f80a79d6a616: Deleting consensus metadata
I20260924 15:59:48.422971 22042 catalog_manager.cc:4841] TS 2b3e38ea389f440ba244f80a79d6a616 (127.16.233.130:41849): tablet cff7260ffc2b408ab3e6527061c0b9d9 (table test-table [id=9f14b649d505406dabdbbbfdf1d56a0d]) successfully deleted
I20260924 15:59:48.424425 22553 ts_tablet_manager.cc:1918] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:59:48.424599 22553 log.cc:1196] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/cff7260ffc2b408ab3e6527061c0b9d9
I20260924 15:59:48.425076 22553 ts_tablet_manager.cc:1939] T cff7260ffc2b408ab3e6527061c0b9d9 P c1255a571ef84d21b42d0df00fe6dbe6: Deleting consensus metadata
I20260924 15:59:48.426748 22043 catalog_manager.cc:4841] TS c1255a571ef84d21b42d0df00fe6dbe6 (127.16.233.131:33019): tablet cff7260ffc2b408ab3e6527061c0b9d9 (table test-table [id=9f14b649d505406dabdbbbfdf1d56a0d]) successfully deleted
I20260924 15:59:48.415760 22552 raft_consensus.cc:2229] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:59:48.512369 22552 raft_consensus.cc:2258] T cff7260ffc2b408ab3e6527061c0b9d9 P ce98631236504b618be45ba32e777b40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:59:48.512574 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 22117
W20260924 15:59:48.529799 22043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.129:33651 (error 108)
I20260924 15:59:48.530623 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 22250
I20260924 15:59:48.531893 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.531383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:33651 (local address 127.0.0.1:56340)
0924 15:59:48.531654 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.531763 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260924 15:59:48.532166 22043 catalog_manager.cc:4584] TS ce98631236504b618be45ba32e777b40 (127.16.233.129:33651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cff7260ffc2b408ab3e6527061c0b9d9: Network error: Client connection negotiation failed: client connection to 127.16.233.129:33651: connect: Connection refused (error 111)
I20260924 15:59:48.533634 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.531528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.16.233.130:41849)
0924 15:59:48.531727 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.531733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.531766 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:59:48.531833 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.531838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.531935 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.532046 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.532995 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.533001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.533006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.533025 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.533065 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.533077 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.533206 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.533300 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.533359 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 15:59:48.534294 22374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:48.585777 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.585173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:33651 (local address 127.0.0.1:56344)
0924 15:59:48.585444 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.585564 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 15:59:48.586912 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.586510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:33651 (local address 127.0.0.1:56358)
0924 15:59:48.586685 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.586771 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 15:59:48.623260 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 22250
I20260924 15:59:48.640401 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 22382
I20260924 15:59:48.643018 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.641207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.16.233.131:33019)
0924 15:59:48.641378 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:48.641383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:48.641401 (+ 18us) server_negotiation.cc:407] Connection header received
0924 15:59:48.641500 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:48.641504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:48.641583 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:48.641685 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:48.642516 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:48.642520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:48.642523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:48.642540 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:48.642571 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:48.642583 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:48.642699 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:48.642733 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 15:59:48.642796 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 15:59:48.643644 22505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:48.647555 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.647065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:33651 (local address 127.0.0.1:56366)
0924 15:59:48.647249 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.647375 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 15:59:48.648711 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.648235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:33651 (local address 127.0.0.1:56374)
0924 15:59:48.648427 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.648529 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 15:59:48.729451 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.728896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:33651 (local address 127.0.0.1:56378)
0924 15:59:48.729140 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.729252 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 15:59:48.730661 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:48.730164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:33651 (local address 127.0.0.1:56394)
0924 15:59:48.730434 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:48.730518 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260924 15:59:48.733196 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 22382
I20260924 15:59:48.750002 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 22026
I20260924 15:59:48.751166 22088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:48.858031 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 22026
2026-09-24T15:59:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2618 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 15:59:48.901985 17318 mini_oidc.cc:152] Starting JWKS server
I20260924 15:59:48.905982 17318 webserver.cc:460] Webserver started at https://127.0.0.1:38277/ using document root <none> and password file <none>
I20260924 15:59:48.906355 17318 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:59:48.907011 17318 webserver.cc:460] Webserver started at http://127.0.0.1:37961/ using document root <none> and password file <none>
2026-09-24T15:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:48Z Disabled control of system clock
I20260924 15:59:48.933720 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:44377
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:44377
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38277/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/testchainca.cert with env {}
W20260924 15:59:49.259696 22566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:49.259999 22566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:49.260092 22566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:49.267400 22566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:49.267516 22566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:49.267565 22566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:49.267622 22566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:49.267668 22566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:59:49.267722 22566 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:59:49.267762 22566 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38277/jwks.json
I20260924 15:59:49.277920 22566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:44377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38277/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:49.279194 22566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:49.280471 22566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:49.292608 22574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:49.292608 22571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:49.292724 22572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:49.295742 22566 server_base.cc:1034] running on GCE node
I20260924 15:59:49.296510 22566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:49.297860 22566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:49.299054 22566 hybrid_clock.cc:648] HybridClock initialized: now 1790265589298993 us; error 51 us; skew 500 ppm
I20260924 15:59:49.299422 22566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 15:59:49.301720 22566 webserver.cc:460] Webserver started at http://127.16.233.190:40675/ using document root <none> and password file <none>
I20260924 15:59:49.302507 22566 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:49.302630 22566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:49.302953 22566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:49.305907 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "5429f4bee4e2468480ade2d7870056ce"
format_stamp: "Formatted at 2026-09-24 15:59:49 on dist-test-slave-xn4h"
server_key: "c50d9ab6e3987bdf8ea69403b1d6d345"
server_key_iv: "30d0ca8157a43fe9810c4ecaa9ec635e"
server_key_version: "encryptionkey@0"
I20260924 15:59:49.306759 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "5429f4bee4e2468480ade2d7870056ce"
format_stamp: "Formatted at 2026-09-24 15:59:49 on dist-test-slave-xn4h"
server_key: "c50d9ab6e3987bdf8ea69403b1d6d345"
server_key_iv: "30d0ca8157a43fe9810c4ecaa9ec635e"
server_key_version: "encryptionkey@0"
I20260924 15:59:49.311866 22566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 15:59:49.315179 22581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:49.316537 22566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 15:59:49.316731 22566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "5429f4bee4e2468480ade2d7870056ce"
format_stamp: "Formatted at 2026-09-24 15:59:49 on dist-test-slave-xn4h"
server_key: "c50d9ab6e3987bdf8ea69403b1d6d345"
server_key_iv: "30d0ca8157a43fe9810c4ecaa9ec635e"
server_key_version: "encryptionkey@0"
I20260924 15:59:49.316964 22566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:49.339928 22566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:49.340813 22566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:49.341132 22566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:49.359411 22566 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:44377
I20260924 15:59:49.359431 22632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:44377 every 8 connection(s)
I20260924 15:59:49.360795 22566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:49.361967 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 22566
I20260924 15:59:49.362205 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:49.363022 17318 external_mini_cluster.cc:1428] Setting key ef27b09cc9b251f5a48cbe299bfcf96f
I20260924 15:59:49.366869 22633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:49.376418 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:49.364803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.16.233.190:44377)
0924 15:59:49.365702 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:49.365714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:49.365755 (+ 41us) server_negotiation.cc:407] Connection header received
0924 15:59:49.367600 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:49.367625 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:49.368081 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:49.368470 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:49.369063 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:49.370463 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:49.371801 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:49.372483 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:49.372687 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:49.373927 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:49.373934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:49.373947 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:49.374567 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:49.374661 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:49.374683 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:49.374832 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:49.375068 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 15:59:49.375386 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":286,"threads_started":1}
I20260924 15:59:49.378052 22633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce: Bootstrap starting.
I20260924 15:59:49.380442 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:59:49.381619 22633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:49.382776 22633 log.cc:824] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:49.385577 22633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce: No bootstrap required, opened a new log
I20260924 15:59:49.393741 22633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5429f4bee4e2468480ade2d7870056ce" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44377 } }
I20260924 15:59:49.394104 22633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:49.394182 22633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5429f4bee4e2468480ade2d7870056ce, State: Initialized, Role: FOLLOWER
I20260924 15:59:49.394755 22633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5429f4bee4e2468480ade2d7870056ce" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44377 } }
I20260924 15:59:49.394950 22633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:49.395048 22633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:49.395181 22633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:49.397675 22633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5429f4bee4e2468480ade2d7870056ce" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44377 } }
I20260924 15:59:49.398150 22633 leader_election.cc:302] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5429f4bee4e2468480ade2d7870056ce; no voters:
I20260924 15:59:49.398761 22633 leader_election.cc:288] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:49.398937 22638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:49.399701 22638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [term 1 LEADER]: Becoming Leader. State: Replica: 5429f4bee4e2468480ade2d7870056ce, State: Running, Role: LEADER
I20260924 15:59:49.400226 22638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5429f4bee4e2468480ade2d7870056ce" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44377 } }
I20260924 15:59:49.400990 22633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:49.403128 22640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5429f4bee4e2468480ade2d7870056ce. Latest consensus state: current_term: 1 leader_uuid: "5429f4bee4e2468480ade2d7870056ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5429f4bee4e2468480ade2d7870056ce" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44377 } } }
I20260924 15:59:49.403617 22640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:49.403226 22639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5429f4bee4e2468480ade2d7870056ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5429f4bee4e2468480ade2d7870056ce" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44377 } } }
I20260924 15:59:49.403878 22639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:49.406004 22647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:49.411283 22647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:49.420151 22647 catalog_manager.cc:1347] Generated new cluster ID: 2daa2e3ed8624da7b1bcdb31645ddc63
I20260924 15:59:49.420290 22647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:49.454674 22647 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:49.456099 22647 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:49.466208 22647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5429f4bee4e2468480ade2d7870056ce: Generated new TSK 0
I20260924 15:59:49.467135 22647 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 15:59:49.511611 22663 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 15:59:49.511870 22663 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 15:59:49.516105 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:49.489095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.16.233.190:44377)
0924 15:59:49.489301 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:49.489308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:49.489750 (+ 442us) server_negotiation.cc:407] Connection header received
0924 15:59:49.490129 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:49.490136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:49.490240 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:49.490442 (+ 202us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:59:49.490977 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:49.492152 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:49.493604 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:49.494341 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:49.494480 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:49.495016 (+ 536us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:59:49.515352 (+ 20336us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 15:59:49.515781 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55214: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 15:59:49.516234 22636 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55214: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 15:59:49.517545 22662 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:59:49.488920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44377 (local address 127.0.0.1:55214)
0924 15:59:49.489636 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:49.489653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:49.489818 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:49.490468 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:49.490471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:49.490482 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:59:49.490819 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:49.490828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:49.492305 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:49.492310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:49.493456 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:49.493468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:49.494513 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:49.494517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:49.494654 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:49.494718 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:59:49.515672 (+ 20954us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 15:59:49.515968 (+ 296us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.233.190:44377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":368,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.233.190:44377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 15:59:49.519461 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 22566
I20260924 15:59:49.520973 22628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:49.618567 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 22566
2026-09-24T15:59:49Z chronyd exiting
I20260924 15:59:49.669669 17318 test_util.cc:175] -----------------------------------------------
I20260924 15:59:49.669826 17318 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265545293861-17318-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (773 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 15:59:49.673771 17318 mini_oidc.cc:152] Starting JWKS server
I20260924 15:59:49.675711 17318 webserver.cc:460] Webserver started at https://127.0.0.1:38001/ using document root <none> and password file <none>
I20260924 15:59:49.675956 17318 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:59:49.676643 17318 webserver.cc:460] Webserver started at http://127.0.0.1:33265/ using document root <none> and password file <none>
2026-09-24T15:59:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:49Z Disabled control of system clock
I20260924 15:59:49.703605 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41319
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:41319
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38001/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/testchainca.cert with env {}
W20260924 15:59:50.029481 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:50.029870 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:50.030045 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:50.037509 22670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:50.037685 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:50.037779 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:50.037887 22670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:50.037977 22670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:59:50.038071 22670 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:59:50.038143 22670 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38001/jwks.json
I20260924 15:59:50.048869 22670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:41319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38001/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:50.050297 22670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:50.051674 22670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:50.064109 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:50.064186 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:50.064139 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:50.070402 22670 server_base.cc:1034] running on GCE node
I20260924 15:59:50.071429 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:50.072985 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:50.074195 22670 hybrid_clock.cc:648] HybridClock initialized: now 1790265590074157 us; error 46 us; skew 500 ppm
I20260924 15:59:50.074604 22670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 15:59:50.077209 22670 webserver.cc:460] Webserver started at http://127.16.233.190:40647/ using document root <none> and password file <none>
I20260924 15:59:50.077956 22670 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:50.078079 22670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:50.078488 22670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:50.081565 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "0d8d8010769d4d559426b7689db028ee"
format_stamp: "Formatted at 2026-09-24 15:59:50 on dist-test-slave-xn4h"
server_key: "41a3a6de506ab471135c2eb1c9fd7559"
server_key_iv: "dc3f0911ace7d106e66d43c010ff1d3b"
server_key_version: "encryptionkey@0"
I20260924 15:59:50.082521 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "0d8d8010769d4d559426b7689db028ee"
format_stamp: "Formatted at 2026-09-24 15:59:50 on dist-test-slave-xn4h"
server_key: "41a3a6de506ab471135c2eb1c9fd7559"
server_key_iv: "dc3f0911ace7d106e66d43c010ff1d3b"
server_key_version: "encryptionkey@0"
I20260924 15:59:50.088012 22670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 15:59:50.091727 22684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:50.093240 22670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:50.093454 22670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "0d8d8010769d4d559426b7689db028ee"
format_stamp: "Formatted at 2026-09-24 15:59:50 on dist-test-slave-xn4h"
server_key: "41a3a6de506ab471135c2eb1c9fd7559"
server_key_iv: "dc3f0911ace7d106e66d43c010ff1d3b"
server_key_version: "encryptionkey@0"
I20260924 15:59:50.093720 22670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:50.105093 22670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:50.105965 22670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:50.106295 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:50.130800 22670 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:41319
I20260924 15:59:50.130708 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:41319 every 8 connection(s)
I20260924 15:59:50.133100 22670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:50.138206 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:50.142686 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 22670
I20260924 15:59:50.142915 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:50.143896 17318 external_mini_cluster.cc:1428] Setting key 6b898cf47a409e5b3976049be3d75f73
I20260924 15:59:50.154222 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee: Bootstrap starting.
I20260924 15:59:50.157584 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:50.158631 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:50.146682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.16.233.190:41319)
0924 15:59:50.147844 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:50.147857 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:50.147907 (+ 50us) server_negotiation.cc:407] Connection header received
0924 15:59:50.149626 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:50.149649 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:50.150138 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:50.150598 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:50.151199 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:50.152562 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:50.153870 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:50.154726 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:50.154929 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:50.156177 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:50.156184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:50.156199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:50.156744 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:50.156835 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:50.156857 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:50.156985 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:50.157185 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 15:59:50.157510 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":541,"threads_started":1}
I20260924 15:59:50.158882 22737 log.cc:824] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:50.162233 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee: No bootstrap required, opened a new log
I20260924 15:59:50.162842 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:59:50.170120 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8d8010769d4d559426b7689db028ee" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41319 } }
I20260924 15:59:50.170625 22737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:50.170779 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8d8010769d4d559426b7689db028ee, State: Initialized, Role: FOLLOWER
I20260924 15:59:50.171439 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8d8010769d4d559426b7689db028ee" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41319 } }
I20260924 15:59:50.171676 22737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:50.171836 22737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:50.172020 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:50.174274 22737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8d8010769d4d559426b7689db028ee" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41319 } }
I20260924 15:59:50.174892 22737 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8d8010769d4d559426b7689db028ee; no voters:
I20260924 15:59:50.175518 22737 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:50.175686 22742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:50.176431 22742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [term 1 LEADER]: Becoming Leader. State: Replica: 0d8d8010769d4d559426b7689db028ee, State: Running, Role: LEADER
I20260924 15:59:50.176965 22742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8d8010769d4d559426b7689db028ee" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41319 } }
I20260924 15:59:50.177778 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:50.179940 22744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8d8010769d4d559426b7689db028ee. Latest consensus state: current_term: 1 leader_uuid: "0d8d8010769d4d559426b7689db028ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8d8010769d4d559426b7689db028ee" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41319 } } }
I20260924 15:59:50.179895 22743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8d8010769d4d559426b7689db028ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8d8010769d4d559426b7689db028ee" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41319 } } }
I20260924 15:59:50.180335 22743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:50.180335 22744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:50.181259 22748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:50.188688 22748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:50.199007 22748 catalog_manager.cc:1347] Generated new cluster ID: e7fdbfdc7e864a9086e9716fec77b3ec
I20260924 15:59:50.199139 22748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:50.222621 22748 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:50.223942 22748 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:50.234970 22748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8d8010769d4d559426b7689db028ee: Generated new TSK 0
I20260924 15:59:50.235867 22748 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 15:59:50.278378 22767 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 15:59:50.278523 22767 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 15:59:50.279649 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:50.250905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.16.233.190:41319)
0924 15:59:50.251148 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:50.251158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:50.251667 (+ 509us) server_negotiation.cc:407] Connection header received
0924 15:59:50.251915 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:50.251919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:50.251992 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:50.252121 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:59:50.252681 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:50.253401 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:50.255456 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:50.255936 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:50.256035 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:50.256647 (+ 612us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:59:50.279058 (+ 22411us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 15:59:50.279427 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51724: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 15:59:50.279793 22740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51724: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 15:59:50.280031 22766 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:59:50.250874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41319 (local address 127.0.0.1:51724)
0924 15:59:50.251548 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:50.251570 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:50.251726 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:50.252169 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:50.252174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:50.252187 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:59:50.252511 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:50.252522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:50.253555 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:50.253561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:50.255305 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:50.255322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:50.256104 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:50.256111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:50.256341 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:50.256391 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:59:50.279323 (+ 22932us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 15:59:50.279617 (+ 294us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.233.190:41319: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":342,"threads_started":1}
I20260924 15:59:50.281817 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 22670
I20260924 15:59:50.283037 22732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:50.377435 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 22670
2026-09-24T15:59:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (759 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 15:59:50.433323 17318 mini_oidc.cc:152] Starting JWKS server
I20260924 15:59:50.435304 17318 webserver.cc:460] Webserver started at https://127.0.0.1:35363/ using document root <none> and password file <none>
I20260924 15:59:50.435565 17318 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:59:50.436194 17318 webserver.cc:460] Webserver started at http://127.0.0.1:38079/ using document root <none> and password file <none>
2026-09-24T15:59:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:50Z Disabled control of system clock
I20260924 15:59:50.464784 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34873
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:34873
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35363/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 15:59:50.789888 22774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:50.790226 22774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:50.790407 22774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:50.797776 22774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:50.797931 22774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:50.798023 22774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:50.798134 22774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:50.798223 22774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:59:50.798339 22774 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:59:50.798400 22774 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35363/jwks.json
I20260924 15:59:50.807885 22774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:34873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35363/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:50.809235 22774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:50.811075 22774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:50.822798 22779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:50.822780 22782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:50.822913 22780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:50.823961 22774 server_base.cc:1034] running on GCE node
I20260924 15:59:50.824729 22774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:50.826218 22774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:50.827477 22774 hybrid_clock.cc:648] HybridClock initialized: now 1790265590827437 us; error 34 us; skew 500 ppm
I20260924 15:59:50.827836 22774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 15:59:50.830370 22774 webserver.cc:460] Webserver started at http://127.16.233.190:37011/ using document root <none> and password file <none>
I20260924 15:59:50.831084 22774 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:50.831243 22774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:50.831593 22774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:50.834585 22774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "3abbbd88d15c4ad093643bf3139f4e0c"
format_stamp: "Formatted at 2026-09-24 15:59:50 on dist-test-slave-xn4h"
server_key: "5518d64c22035453ad1aee11b22bab04"
server_key_iv: "7825f6cb287db681a479e043a71673a4"
server_key_version: "encryptionkey@0"
I20260924 15:59:50.835434 22774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "3abbbd88d15c4ad093643bf3139f4e0c"
format_stamp: "Formatted at 2026-09-24 15:59:50 on dist-test-slave-xn4h"
server_key: "5518d64c22035453ad1aee11b22bab04"
server_key_iv: "7825f6cb287db681a479e043a71673a4"
server_key_version: "encryptionkey@0"
I20260924 15:59:50.840862 22774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 15:59:50.844436 22788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:50.845772 22774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 15:59:50.846062 22774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "3abbbd88d15c4ad093643bf3139f4e0c"
format_stamp: "Formatted at 2026-09-24 15:59:50 on dist-test-slave-xn4h"
server_key: "5518d64c22035453ad1aee11b22bab04"
server_key_iv: "7825f6cb287db681a479e043a71673a4"
server_key_version: "encryptionkey@0"
I20260924 15:59:50.846290 22774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:50.862581 22774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:50.863448 22774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:50.863775 22774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:50.882524 22774 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:34873
I20260924 15:59:50.882543 22839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:34873 every 8 connection(s)
I20260924 15:59:50.883999 22774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:50.884505 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 22774
I20260924 15:59:50.884737 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:50.885586 17318 external_mini_cluster.cc:1428] Setting key 7f32fc6608297e798730c43b9801812e
I20260924 15:59:50.888447 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:50.899788 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:50.887645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.16.233.190:34873)
0924 15:59:50.888709 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:50.888724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:50.888769 (+ 45us) server_negotiation.cc:407] Connection header received
0924 15:59:50.890438 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:50.890460 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:50.890928 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:50.891287 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:50.891905 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:50.893255 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:50.895169 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:50.895849 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:50.895986 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:50.897256 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:50.897264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:50.897277 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:50.897828 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:50.897926 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:50.897955 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:50.898046 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:50.898351 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 15:59:50.898708 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":365,"threads_started":1}
I20260924 15:59:50.901347 22840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c: Bootstrap starting.
I20260924 15:59:50.904493 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:59:50.905228 22840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:50.906245 22840 log.cc:824] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:50.909334 22840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c: No bootstrap required, opened a new log
I20260924 15:59:50.916898 22840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abbbd88d15c4ad093643bf3139f4e0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34873 } }
I20260924 15:59:50.917384 22840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:50.917508 22840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3abbbd88d15c4ad093643bf3139f4e0c, State: Initialized, Role: FOLLOWER
I20260924 15:59:50.918084 22840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abbbd88d15c4ad093643bf3139f4e0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34873 } }
I20260924 15:59:50.918365 22840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:50.918514 22840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:50.918697 22840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:50.920833 22840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abbbd88d15c4ad093643bf3139f4e0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34873 } }
I20260924 15:59:50.921351 22840 leader_election.cc:302] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3abbbd88d15c4ad093643bf3139f4e0c; no voters:
I20260924 15:59:50.921990 22840 leader_election.cc:288] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:50.922202 22845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:50.923465 22845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [term 1 LEADER]: Becoming Leader. State: Replica: 3abbbd88d15c4ad093643bf3139f4e0c, State: Running, Role: LEADER
I20260924 15:59:50.923964 22845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abbbd88d15c4ad093643bf3139f4e0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34873 } }
I20260924 15:59:50.924417 22840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:50.927018 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3abbbd88d15c4ad093643bf3139f4e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abbbd88d15c4ad093643bf3139f4e0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34873 } } }
I20260924 15:59:50.926967 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3abbbd88d15c4ad093643bf3139f4e0c. Latest consensus state: current_term: 1 leader_uuid: "3abbbd88d15c4ad093643bf3139f4e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abbbd88d15c4ad093643bf3139f4e0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34873 } } }
I20260924 15:59:50.927650 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:50.927733 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:50.928551 22854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:50.937981 22854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:50.948088 22854 catalog_manager.cc:1347] Generated new cluster ID: 0ea047780d8d4d13a2cc98e7774f8d08
I20260924 15:59:50.948237 22854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:50.964169 22854 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:50.966045 22854 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:50.979498 22854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3abbbd88d15c4ad093643bf3139f4e0c: Generated new TSK 0
I20260924 15:59:50.980376 22854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:59:51.013017 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:51.004343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.16.233.190:34873)
0924 15:59:51.004562 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:51.004569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:51.004969 (+ 400us) server_negotiation.cc:407] Connection header received
0924 15:59:51.005196 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:51.005204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:51.005315 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:51.005486 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:59:51.005984 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.007156 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.008552 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.009291 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.009419 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:51.009950 (+ 531us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:59:51.012345 (+ 2395us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 15:59:51.012693 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 15:59:51.013149 22843 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 15:59:51.015136 22870 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:59:51.004146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:34873 (local address 127.0.0.1:40782)
0924 15:59:51.004872 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:51.004888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:51.005007 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:51.005523 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:51.005527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:51.005538 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:59:51.005833 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:51.005841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.007305 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.007310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:51.008407 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:51.008419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.009474 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.009478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:51.009617 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:51.009652 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:59:51.012550 (+ 2898us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 15:59:51.014696 (+ 2146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.233.190:34873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":1808,"thread_start_us":436,"threads_started":1}
I20260924 15:59:51.016180 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 22774
I20260924 15:59:51.017719 22835 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 15:59:51.025393 22871 webserver.cc:533] Webserver: sslize failed:
E20260924 15:59:51.025482 22871 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 15:59:51.113683 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 22774
2026-09-24T15:59:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (736 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T15:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:51Z Disabled control of system clock
I20260924 15:59:51.198428 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41429
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:41429
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/testchainca.cert with env {}
W20260924 15:59:51.527429 22876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:51.527812 22876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:51.527964 22876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:51.535422 22876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:51.535591 22876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:51.535686 22876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:51.535799 22876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:51.535876 22876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:59:51.535964 22876 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:59:51.536042 22876 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 15:59:51.546468 22876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:41429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:51.547902 22876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:51.549234 22876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:51.561379 22881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:51.561414 22882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:51.561285 22884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:51.562480 22876 server_base.cc:1034] running on GCE node
I20260924 15:59:51.563375 22876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:51.564854 22876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:51.566066 22876 hybrid_clock.cc:648] HybridClock initialized: now 1790265591566017 us; error 39 us; skew 500 ppm
I20260924 15:59:51.566501 22876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 15:59:51.568897 22876 webserver.cc:460] Webserver started at http://127.16.233.190:45761/ using document root <none> and password file <none>
I20260924 15:59:51.569615 22876 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:51.569736 22876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:51.570118 22876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:51.573192 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "25a942fffe3c457bbf8de0188b523407"
format_stamp: "Formatted at 2026-09-24 15:59:51 on dist-test-slave-xn4h"
server_key: "c36ce31e070df8ac6de0f2c7f8dce9fd"
server_key_iv: "f4b4adc46d62d8419641314728364d54"
server_key_version: "encryptionkey@0"
I20260924 15:59:51.574079 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "25a942fffe3c457bbf8de0188b523407"
format_stamp: "Formatted at 2026-09-24 15:59:51 on dist-test-slave-xn4h"
server_key: "c36ce31e070df8ac6de0f2c7f8dce9fd"
server_key_iv: "f4b4adc46d62d8419641314728364d54"
server_key_version: "encryptionkey@0"
I20260924 15:59:51.579286 22876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 15:59:51.582922 22890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:51.584311 22876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:51.584519 22876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "25a942fffe3c457bbf8de0188b523407"
format_stamp: "Formatted at 2026-09-24 15:59:51 on dist-test-slave-xn4h"
server_key: "c36ce31e070df8ac6de0f2c7f8dce9fd"
server_key_iv: "f4b4adc46d62d8419641314728364d54"
server_key_version: "encryptionkey@0"
I20260924 15:59:51.584756 22876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:51.599601 22876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:51.600548 22876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:51.600857 22876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:51.619719 22876 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:41429
I20260924 15:59:51.619725 22941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:41429 every 8 connection(s)
I20260924 15:59:51.621196 22876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:51.625015 22942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:51.628634 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 22876
I20260924 15:59:51.628832 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:51.629678 17318 external_mini_cluster.cc:1428] Setting key e946c9342d27d28647cad8edd2f6c3d7
I20260924 15:59:51.638043 22942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407: Bootstrap starting.
I20260924 15:59:51.641269 22942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:51.642463 22942 log.cc:824] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:51.643607 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:51.631707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.16.233.190:41429)
0924 15:59:51.632731 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:51.632743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:51.632795 (+ 52us) server_negotiation.cc:407] Connection header received
0924 15:59:51.634603 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:51.634631 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:51.635165 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:51.635620 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:51.636238 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.637488 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.638976 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.639681 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.639893 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:51.641112 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:51.641120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:51.641141 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:59:51.641775 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:51.641868 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:59:51.641889 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:51.642017 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:51.642263 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 15:59:51.642617 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":336,"threads_started":1}
I20260924 15:59:51.645756 22942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407: No bootstrap required, opened a new log
I20260924 15:59:51.647630 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:59:51.654117 22942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a942fffe3c457bbf8de0188b523407" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41429 } }
I20260924 15:59:51.654608 22942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:51.654739 22942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a942fffe3c457bbf8de0188b523407, State: Initialized, Role: FOLLOWER
I20260924 15:59:51.655355 22942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a942fffe3c457bbf8de0188b523407" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41429 } }
I20260924 15:59:51.655580 22942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:51.655748 22942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:51.655918 22942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:51.658699 22942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a942fffe3c457bbf8de0188b523407" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41429 } }
I20260924 15:59:51.659232 22942 leader_election.cc:302] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25a942fffe3c457bbf8de0188b523407; no voters:
I20260924 15:59:51.659904 22942 leader_election.cc:288] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:51.660025 22947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:51.661334 22947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [term 1 LEADER]: Becoming Leader. State: Replica: 25a942fffe3c457bbf8de0188b523407, State: Running, Role: LEADER
I20260924 15:59:51.661829 22947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a942fffe3c457bbf8de0188b523407" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41429 } }
I20260924 15:59:51.662257 22942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:51.664651 22949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25a942fffe3c457bbf8de0188b523407. Latest consensus state: current_term: 1 leader_uuid: "25a942fffe3c457bbf8de0188b523407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a942fffe3c457bbf8de0188b523407" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41429 } } }
I20260924 15:59:51.665005 22949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:51.664754 22948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25a942fffe3c457bbf8de0188b523407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a942fffe3c457bbf8de0188b523407" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41429 } } }
I20260924 15:59:51.665175 22948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:51.672158 22954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:51.676553 22954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:51.685289 22954 catalog_manager.cc:1347] Generated new cluster ID: e877e0a3111c4b5b9bf3e62a3ec83a20
I20260924 15:59:51.685421 22954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:51.710518 22954 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:51.712625 22954 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:51.728111 22954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25a942fffe3c457bbf8de0188b523407: Generated new TSK 0
I20260924 15:59:51.729274 22954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:59:51.855187 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:51.753079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.16.233.190:41429)
0924 15:59:51.753313 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:51.753319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:51.753687 (+ 368us) server_negotiation.cc:407] Connection header received
0924 15:59:51.753991 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:51.753996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:51.754066 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:51.754194 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:59:51.754745 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.755945 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.757985 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.758464 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.758558 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:51.759160 (+ 602us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:59:51.854408 (+ 95248us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 15:59:51.854846 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45528: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":95}
W20260924 15:59:51.855307 22971 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:59:51.752874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41429 (local address 127.0.0.1:45528)
0924 15:59:51.753549 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:51.753569 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:51.753764 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:51.754243 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:51.754250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:51.754265 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:59:51.754571 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:51.754585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.756097 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.756103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:51.757841 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:51.757857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:51.758634 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:51.758652 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:51.758839 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:51.758889 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:59:51.854706 (+ 95817us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 15:59:51.854960 (+ 254us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.233.190:41429: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":323,"threads_started":1}
W20260924 15:59:51.855335 22945 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45528: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260924 15:59:51.856683 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 22876
I20260924 15:59:51.858144 22937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:51.964542 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 22876
2026-09-24T15:59:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (830 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (424 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (414 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:52 dist-test-slave-xn4h krb5kdc[22986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:52 dist-test-slave-xn4h krb5kdc[22986](info): set up 2 sockets
Sep 24 15:59:52 dist-test-slave-xn4h krb5kdc[22986](info): commencing operation
krb5kdc: starting...
W20260924 15:59:53.904686 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:54 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 15:59:54.142248 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:38491
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:54.480792 22999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:54.481123 22999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:54.481293 22999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:54.489068 22999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:54.489209 22999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:54.489257 22999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:54.489382 22999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:54.489499 22999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:54.500000 22999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:54.501521 22999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:54.502966 22999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:54.515843 23004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:54.515940 23005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:54.515894 23007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:54.516953 22999 server_base.cc:1034] running on GCE node
I20260924 15:59:54.517765 22999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:54.519392 22999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:54.520622 22999 hybrid_clock.cc:648] HybridClock initialized: now 1790265594520561 us; error 49 us; skew 500 ppm
I20260924 15:59:54.520967 22999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:54 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265594, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:54.524387 22999 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 15:59:54.526137 22999 webserver.cc:460] Webserver started at http://127.16.233.190:40391/ using document root <none> and password file <none>
I20260924 15:59:54.527051 22999 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:54.527180 22999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:54.527474 22999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:54.530730 22999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "0eea673363e845f48fc77a3d5f324279"
format_stamp: "Formatted at 2026-09-24 15:59:54 on dist-test-slave-xn4h"
server_key: "0d067fbdfa2f6f46ea0aee998da31773"
server_key_iv: "bdc9dafcf46170443fd588691eccddad"
server_key_version: "encryptionkey@0"
I20260924 15:59:54.531545 22999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "0eea673363e845f48fc77a3d5f324279"
format_stamp: "Formatted at 2026-09-24 15:59:54 on dist-test-slave-xn4h"
server_key: "0d067fbdfa2f6f46ea0aee998da31773"
server_key_iv: "bdc9dafcf46170443fd588691eccddad"
server_key_version: "encryptionkey@0"
I20260924 15:59:54.537252 22999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 15:59:54.541172 23014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:54.542773 22999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:54.542930 22999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "0eea673363e845f48fc77a3d5f324279"
format_stamp: "Formatted at 2026-09-24 15:59:54 on dist-test-slave-xn4h"
server_key: "0d067fbdfa2f6f46ea0aee998da31773"
server_key_iv: "bdc9dafcf46170443fd588691eccddad"
server_key_version: "encryptionkey@0"
I20260924 15:59:54.543113 22999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:54.556880 22999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:54.561447 22999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:54.561717 22999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:54.581436 22999 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:38491
I20260924 15:59:54.581467 23065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:38491 every 8 connection(s)
I20260924 15:59:54.582983 22999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:54.585005 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 22999
I20260924 15:59:54.585275 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:54.586186 17318 external_mini_cluster.cc:1428] Setting key 272c5597d005456cc020c4b3a7893d59
I20260924 15:59:54.588109 23066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 15:59:54 dist-test-slave-xn4h krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:54.601279 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279: Bootstrap starting.
I20260924 15:59:54.605087 23066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:54.606285 23066 log.cc:824] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:54.610260 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279: No bootstrap required, opened a new log
I20260924 15:59:54.610962 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:54.588608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.16.233.190:38491)
0924 15:59:54.589452 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:54.589464 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:54.589509 (+ 45us) server_negotiation.cc:407] Connection header received
0924 15:59:54.591356 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:54.591383 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:54.591844 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:54.592290 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:54.593861 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:54.595368 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:54.596692 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:54.597472 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:54.597706 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:54.602277 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:54.602286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:54.602357 (+ 71us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:54.602422 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:54.605662 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:54.606595 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:54.606603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:54.606610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:54.606797 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:54.607215 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:54.607221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:54.607224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:54.608087 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:54.608369 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:54.608911 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 15:59:54.609687 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":363,"threads_started":1}
I20260924 15:59:54.618947 23066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea673363e845f48fc77a3d5f324279" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38491 } }
I20260924 15:59:54.619406 23066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:54.619520 23066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eea673363e845f48fc77a3d5f324279, State: Initialized, Role: FOLLOWER
I20260924 15:59:54.620206 23066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea673363e845f48fc77a3d5f324279" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38491 } }
I20260924 15:59:54.620474 23066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:54.620630 23066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:54.620823 23066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:54.623245 23066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea673363e845f48fc77a3d5f324279" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38491 } }
I20260924 15:59:54.623893 23066 leader_election.cc:302] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eea673363e845f48fc77a3d5f324279; no voters:
I20260924 15:59:54.624572 23066 leader_election.cc:288] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:54.624716 23071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:54.625494 23071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [term 1 LEADER]: Becoming Leader. State: Replica: 0eea673363e845f48fc77a3d5f324279, State: Running, Role: LEADER
I20260924 15:59:54.626013 23071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea673363e845f48fc77a3d5f324279" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38491 } }
I20260924 15:59:54.626883 23066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:54.628947 23073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eea673363e845f48fc77a3d5f324279. Latest consensus state: current_term: 1 leader_uuid: "0eea673363e845f48fc77a3d5f324279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea673363e845f48fc77a3d5f324279" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38491 } } }
I20260924 15:59:54.629402 23073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:54.630272 23072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eea673363e845f48fc77a3d5f324279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea673363e845f48fc77a3d5f324279" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38491 } } }
I20260924 15:59:54.630631 23072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:54.635881 23076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:54.641985 23076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:54.651384 23076 catalog_manager.cc:1347] Generated new cluster ID: 9f16ab2a764f42e2a27ff41e3954c0ed
I20260924 15:59:54.651517 23076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:54.684217 23076 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:54.685700 23076 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:54.698458 23076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eea673363e845f48fc77a3d5f324279: Generated new TSK 0
I20260924 15:59:54.699411 23076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 15:59:54.755410 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38491
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:55.098217 23092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:55.098623 23092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:55.098809 23092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:55.106550 23092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:55.106667 23092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:55.106873 23092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:55.117595 23092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:55.119060 23092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:55.120456 23092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:55.134291 23100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:55.134351 23098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:55.134374 23097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:55.135603 23092 server_base.cc:1034] running on GCE node
I20260924 15:59:55.136368 23092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:55.137444 23092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:55.138681 23092 hybrid_clock.cc:648] HybridClock initialized: now 1790265595138607 us; error 65 us; skew 500 ppm
I20260924 15:59:55.139025 23092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:55 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265595, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:55.142437 23092 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 15:59:55.144013 23092 webserver.cc:460] Webserver started at http://127.16.233.129:33493/ using document root <none> and password file <none>
I20260924 15:59:55.144748 23092 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:55.144907 23092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:55.145313 23092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:55.148672 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "d2d293d9750b45b0b86ed7e8eaddc7b6"
format_stamp: "Formatted at 2026-09-24 15:59:55 on dist-test-slave-xn4h"
server_key: "7594d9c124d98f1954890d94451ef930"
server_key_iv: "86a6ee9cd516af65a88f8b8e15853a00"
server_key_version: "encryptionkey@0"
I20260924 15:59:55.149546 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "d2d293d9750b45b0b86ed7e8eaddc7b6"
format_stamp: "Formatted at 2026-09-24 15:59:55 on dist-test-slave-xn4h"
server_key: "7594d9c124d98f1954890d94451ef930"
server_key_iv: "86a6ee9cd516af65a88f8b8e15853a00"
server_key_version: "encryptionkey@0"
I20260924 15:59:55.154973 23092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 15:59:55.158613 23107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:55.159927 23092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:55.160115 23092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "d2d293d9750b45b0b86ed7e8eaddc7b6"
format_stamp: "Formatted at 2026-09-24 15:59:55 on dist-test-slave-xn4h"
server_key: "7594d9c124d98f1954890d94451ef930"
server_key_iv: "86a6ee9cd516af65a88f8b8e15853a00"
server_key_version: "encryptionkey@0"
I20260924 15:59:55.160365 23092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:55.175310 23092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:55.179839 23092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:55.180171 23092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:55.181201 23092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:55.182687 23092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:55.182793 23092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:55.182950 23092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:55.183027 23092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:55.217083 23092 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:37567
I20260924 15:59:55.217146 23221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:37567 every 8 connection(s)
I20260924 15:59:55.218562 23092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:55.226989 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 23092
I20260924 15:59:55.227232 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:55.227927 17318 external_mini_cluster.cc:1428] Setting key 5fbef3eb0ef3a5337ea327be6f34d31a
Sep 24 15:59:55 dist-test-slave-xn4h krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265595, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:55.240754 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:55.221372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:60893 (local address 127.16.233.190:38491)
0924 15:59:55.221870 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:55.221874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:55.222833 (+ 959us) server_negotiation.cc:407] Connection header received
0924 15:59:55.224738 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:55.224744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:55.224847 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:55.224969 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:55.226603 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:55.227497 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:55.229491 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:55.230060 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:55.230191 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:55.233860 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:55.233870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:55.233878 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:55.233927 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:55.237286 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:55.238343 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:55.238355 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:55.238358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:55.238483 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:55.238968 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:55.238971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:55.238973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:55.239253 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:55.239362 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:55.240305 (+ 943us) server_negotiation.cc:299] Negotiation successful
0924 15:59:55.240481 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":263,"threads_started":1}
I20260924 15:59:55.242239 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:55.221797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:38491 (local address 127.16.233.129:60893)
0924 15:59:55.222659 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:55.222699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:55.224443 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:55.225221 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:55.225230 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:55.225755 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:55.226423 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:55.226443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:55.227675 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:55.227679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:55.229355 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:55.229370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:55.230291 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:55.230296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:55.230550 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:55.231243 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:55.231265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:55.233622 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:55.237445 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:55.237459 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:55.237479 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:55.238163 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:55.238588 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:55.238596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:55.238603 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:55.238853 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:55.239385 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:55.239395 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:55.239874 (+ 479us) client_negotiation.cc:769] Sending connection context
0924 15:59:55.240190 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 15:59:55.240785 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":334,"threads_started":1}
I20260924 15:59:55.243975 23222 heartbeater.cc:347] Connected to a master server at 127.16.233.190:38491
I20260924 15:59:55.244323 23222 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:55.245110 23222 heartbeater.cc:511] Master 127.16.233.190:38491 requested a full tablet report, sending...
I20260924 15:59:55.247403 23031 ts_manager.cc:194] Registered new tserver with Master: d2d293d9750b45b0b86ed7e8eaddc7b6 (127.16.233.129:37567)
I20260924 15:59:55.248863 23031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:60893
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 15:59:55.277334 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38491
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:55.616132 23229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:55.616498 23229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:55.616664 23229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:55.624212 23229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:55.624404 23229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:55.624600 23229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:55.634791 23229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:55.636277 23229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:55.637742 23229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:55.651593 23234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:55.651643 23235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:55.651687 23237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:55.652503 23229 server_base.cc:1034] running on GCE node
I20260924 15:59:55.653244 23229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:55.654342 23229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:55.655576 23229 hybrid_clock.cc:648] HybridClock initialized: now 1790265595655530 us; error 43 us; skew 500 ppm
I20260924 15:59:55.655933 23229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:55 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265595, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:55.659279 23229 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 15:59:55.660880 23229 webserver.cc:460] Webserver started at http://127.16.233.130:34501/ using document root <none> and password file <none>
I20260924 15:59:55.661628 23229 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:55.661791 23229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:55.662120 23229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:55.665478 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "6e628f168e9a42d983356d671a7ad0aa"
format_stamp: "Formatted at 2026-09-24 15:59:55 on dist-test-slave-xn4h"
server_key: "ebb5ee1b130f483a2693af6fed45ae05"
server_key_iv: "d685a3c4579ff05499ffc9afed4ff9e4"
server_key_version: "encryptionkey@0"
I20260924 15:59:55.666389 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "6e628f168e9a42d983356d671a7ad0aa"
format_stamp: "Formatted at 2026-09-24 15:59:55 on dist-test-slave-xn4h"
server_key: "ebb5ee1b130f483a2693af6fed45ae05"
server_key_iv: "d685a3c4579ff05499ffc9afed4ff9e4"
server_key_version: "encryptionkey@0"
I20260924 15:59:55.671923 23229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 15:59:55.675624 23244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:55.677079 23229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 15:59:55.677286 23229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "6e628f168e9a42d983356d671a7ad0aa"
format_stamp: "Formatted at 2026-09-24 15:59:55 on dist-test-slave-xn4h"
server_key: "ebb5ee1b130f483a2693af6fed45ae05"
server_key_iv: "d685a3c4579ff05499ffc9afed4ff9e4"
server_key_version: "encryptionkey@0"
I20260924 15:59:55.677515 23229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:55.699314 23229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:55.704238 23229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:55.704582 23229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:55.705638 23229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:55.707176 23229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:55.707289 23229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:55.707440 23229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:55.707512 23229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:55.742566 23229 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:37661
I20260924 15:59:55.742655 23357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:37661 every 8 connection(s)
I20260924 15:59:55.743975 23229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:55.748518 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 23229
I20260924 15:59:55.748708 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:55.749202 17318 external_mini_cluster.cc:1428] Setting key c19fc431392562100cb98545c76f842f
Sep 24 15:59:55 dist-test-slave-xn4h krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265595, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:55.766852 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:55.746924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:36809 (local address 127.16.233.190:38491)
0924 15:59:55.747445 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:55.747450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:55.748423 (+ 973us) server_negotiation.cc:407] Connection header received
0924 15:59:55.750958 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:55.750962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:55.751029 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:55.751195 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:55.752872 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:55.753634 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:55.755313 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:55.755949 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:55.756082 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:55.760446 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:55.760452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:55.760455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:55.760492 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:55.763284 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:55.764131 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:55.764136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:55.764138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:55.764296 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:55.764932 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:55.764935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:55.764937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:55.765216 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:55.765387 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:55.766156 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 15:59:55.766438 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":290,"threads_started":1}
I20260924 15:59:55.767895 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:55.747463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:38491 (local address 127.16.233.130:36809)
0924 15:59:55.748236 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:55.748281 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:55.750634 (+ 2353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:55.751503 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:55.751516 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:55.752068 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:55.752685 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:55.752704 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:55.753798 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:55.753803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:55.755158 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:55.755170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:55.756325 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:55.756336 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:55.756754 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:55.757549 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:55.757580 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:55.760118 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:55.763458 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:55.763469 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:55.763489 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:55.763980 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:55.764446 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:55.764454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:55.764462 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:55.764806 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:55.765353 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:55.765362 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:55.765760 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 15:59:55.766055 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 15:59:55.766692 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":285,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 15:59:55.769680 23358 heartbeater.cc:347] Connected to a master server at 127.16.233.190:38491
I20260924 15:59:55.770022 23358 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:55.770939 23358 heartbeater.cc:511] Master 127.16.233.190:38491 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
I20260924 15:59:55.772641 23031 ts_manager.cc:194] Registered new tserver with Master: 6e628f168e9a42d983356d671a7ad0aa (127.16.233.130:37661)
I20260924 15:59:55.773540 23031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:36809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 15:59:55.799749 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38491
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:56.144398 23364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:56.144711 23364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:56.144809 23364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:56.152642 23364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:56.152769 23364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:56.152889 23364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 15:59:56.163094 23364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:56.164479 23364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:56.165910 23364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:56.181097 23372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:56.181049 23370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:56.181140 23369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:56.182042 23364 server_base.cc:1034] running on GCE node
I20260924 15:59:56.182946 23364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:56.183938 23364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:56.185171 23364 hybrid_clock.cc:648] HybridClock initialized: now 1790265596185114 us; error 52 us; skew 500 ppm
I20260924 15:59:56.185515 23364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:56.188705 23364 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 15:59:56.190430 23364 webserver.cc:460] Webserver started at http://127.16.233.131:33393/ using document root <none> and password file <none>
I20260924 15:59:56.191205 23364 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:56.191313 23364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:56.191586 23364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:56.194630 23364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "fe9b704a0d024a2ebfa69dd37fa6528a"
format_stamp: "Formatted at 2026-09-24 15:59:56 on dist-test-slave-xn4h"
server_key: "f5838154f2e7d363877d26482c8193a7"
server_key_iv: "a67c5dbf94d365d86c4163353ce30af6"
server_key_version: "encryptionkey@0"
I20260924 15:59:56.195384 23364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "fe9b704a0d024a2ebfa69dd37fa6528a"
format_stamp: "Formatted at 2026-09-24 15:59:56 on dist-test-slave-xn4h"
server_key: "f5838154f2e7d363877d26482c8193a7"
server_key_iv: "a67c5dbf94d365d86c4163353ce30af6"
server_key_version: "encryptionkey@0"
I20260924 15:59:56.200754 23364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 15:59:56.204282 23379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:56.205641 23364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 15:59:56.205822 23364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "fe9b704a0d024a2ebfa69dd37fa6528a"
format_stamp: "Formatted at 2026-09-24 15:59:56 on dist-test-slave-xn4h"
server_key: "f5838154f2e7d363877d26482c8193a7"
server_key_iv: "a67c5dbf94d365d86c4163353ce30af6"
server_key_version: "encryptionkey@0"
I20260924 15:59:56.206097 23364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:56.238762 23364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:56.243072 23364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:56.243322 23364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:56.244256 23364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:56.245604 23364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:56.245677 23364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:56.245761 23364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:56.245812 23364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:56.252005 23222 heartbeater.cc:503] Master 127.16.233.190:38491 was elected leader, sending a full tablet report...
I20260924 15:59:56.280581 23364 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:34851
I20260924 15:59:56.280625 23492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:34851 every 8 connection(s)
I20260924 15:59:56.282217 23364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 15:59:56.292487 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 23364
I20260924 15:59:56.292713 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 15:59:56.293390 17318 external_mini_cluster.cc:1428] Setting key dfa9ab7ed8cdf949ad570c6206abb98d
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 15:59:56.304870 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:56.284911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:50011 (local address 127.16.233.190:38491)
0924 15:59:56.285414 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.285420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.286507 (+ 1087us) server_negotiation.cc:407] Connection header received
0924 15:59:56.288407 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.288411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.288489 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.288594 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.290276 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.291284 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.293613 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.294214 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.294398 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:56.298618 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:56.298624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:56.298627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:56.298665 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:56.301447 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.302294 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.302300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.302337 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.302482 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.302961 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.302968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.302971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.303277 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:56.303449 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:56.304226 (+ 777us) server_negotiation.cc:299] Negotiation successful
0924 15:59:56.304468 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":238,"threads_started":1}
I20260924 15:59:56.305886 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:56.285525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:38491 (local address 127.16.233.131:50011)
0924 15:59:56.286263 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:56.286384 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:56.288097 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:56.288853 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:56.288862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:56.289405 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:56.290026 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:56.290046 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.291462 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.291466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:56.293409 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:56.293427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.294477 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.294487 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:56.294798 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:56.295650 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:56.295678 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:56.298364 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:56.301640 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:56.301651 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:56.301687 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:56.302151 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:56.302604 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:56.302610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:56.302614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:56.302841 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:56.303413 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:56.303422 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:56.303831 (+ 409us) client_negotiation.cc:769] Sending connection context
0924 15:59:56.304114 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 15:59:56.304710 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":320,"threads_started":1}
I20260924 15:59:56.307530 23493 heartbeater.cc:347] Connected to a master server at 127.16.233.190:38491
I20260924 15:59:56.307857 23493 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:56.308827 23493 heartbeater.cc:511] Master 127.16.233.190:38491 requested a full tablet report, sending...
I20260924 15:59:56.310482 23031 ts_manager.cc:194] Registered new tserver with Master: fe9b704a0d024a2ebfa69dd37fa6528a (127.16.233.131:34851)
I20260924 15:59:56.311385 23031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:50011
I20260924 15:59:56.321763 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/test-admin.keytab.
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:56.348614 17318 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 15:59:56.354187 23501 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:59:56.351592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.16.233.129:37567)
0924 15:59:56.352259 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.352266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.352292 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:59:56.352353 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.352360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.352578 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.352751 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.353662 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35244: BlockingRecv error: recv got EOF from 127.0.0.1:35244 (error 108)
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":339,"threads_started":1}
W20260924 15:59:56.354530 17318 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5cc)
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:56.356235 17318 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:56.357709 17318 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:56.374243 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:56.359559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.16.233.129:37567)
0924 15:59:56.359823 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.359829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.360043 (+ 214us) server_negotiation.cc:407] Connection header received
0924 15:59:56.360189 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.360194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.360260 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.360348 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.361383 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.362463 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.363773 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.364460 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.364627 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:56.368367 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:56.368389 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:56.368404 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:56.368453 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:56.371481 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.372140 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.372147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.372149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.372269 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.372599 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.372602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.372604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.373010 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:56.373231 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:56.373649 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 15:59:56.373899 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 15:59:56.380327 23501 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:59:56.378408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.16.233.129:37567)
0924 15:59:56.378568 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.378572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.378988 (+ 416us) server_negotiation.cc:407] Connection header received
0924 15:59:56.379114 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.379118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.379188 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.379270 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.380015 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35272: BlockingRecv error: recv got EOF from 127.0.0.1:35272 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 15:59:56.380538 17318 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5cc' of type FILE
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:56.382215 17318 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:56.383625 17318 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:56.398487 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:56.385028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.16.233.129:37567)
0924 15:59:56.385261 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.385268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.385504 (+ 236us) server_negotiation.cc:407] Connection header received
0924 15:59:56.385652 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.385656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.385735 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.385821 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.386898 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.387601 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.388778 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.389271 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.389394 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:56.393114 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:56.393119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:56.393123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:56.393175 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:56.396073 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.396670 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.396674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.396676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.396805 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.397147 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.397150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.397151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.397428 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:56.397647 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:56.397865 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 15:59:56.398043 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 15:59:56.402976 23501 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:59:56.401302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.16.233.129:37567)
0924 15:59:56.401524 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.401528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.401610 (+ 82us) server_negotiation.cc:407] Connection header received
0924 15:59:56.401750 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.401753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.401837 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.401922 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.402704 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35286: BlockingRecv error: recv got EOF from 127.0.0.1:35286 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 15:59:56.403051 17318 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265545293861-17318-0/krb5kdc/krb5cc)
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:56.404680 17318 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:56.406064 17318 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 15:59:56.421402 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:56.407569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.16.233.129:37567)
0924 15:59:56.407900 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.407906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.408098 (+ 192us) server_negotiation.cc:407] Connection header received
0924 15:59:56.408242 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.408246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.408322 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.408436 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.409408 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.410247 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.411651 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.412079 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.412190 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:56.416335 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:56.416341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:56.416345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:56.416386 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:56.418926 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.419585 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.419589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.419591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.419756 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.420078 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.420081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.420083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.420373 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:56.420602 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:56.420837 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 15:59:56.421041 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260924 15:59:56.423264 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 23092
I20260924 15:59:56.435581 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:56.423975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.16.233.129:37567)
0924 15:59:56.424106 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.424110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.424532 (+ 422us) server_negotiation.cc:407] Connection header received
0924 15:59:56.424645 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.424650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.424711 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.424790 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.425771 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.426587 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.427925 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.428352 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.428462 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:56.430522 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:56.430528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:56.430533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:56.430582 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:56.433180 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.433823 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.433826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.433829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.433964 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.434279 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.434282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.434284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.434604 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:56.434806 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:56.435112 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 15:59:56.435292 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:59:56.436205 23217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:56.523916 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 23092
I20260924 15:59:56.541163 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 23229
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 15:59:56.561018 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:56.542101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.16.233.130:37661)
0924 15:59:56.542739 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.542747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.542772 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:59:56.542834 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.542841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.543050 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.543181 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.544128 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.545193 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.546550 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.547199 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.547367 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:56.555095 (+ 7728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:56.555115 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:56.555132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:56.555196 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:56.558050 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.558901 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.558909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.558912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.559052 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.559468 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.559472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.559474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.559867 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:56.560108 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:56.560404 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 15:59:56.560702 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":343,"threads_started":1}
I20260924 15:59:56.562294 23353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:56.647640 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 23229
I20260924 15:59:56.664044 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 23364
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 15:59:56.680025 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:56.665162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.16.233.131:34851)
0924 15:59:56.665891 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:56.665900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:56.665929 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:59:56.665990 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:56.665996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:56.666212 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:56.666431 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:56.667422 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.668461 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.669677 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:56.670243 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:56.670442 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:56.674383 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:56.674406 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:56.674422 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:56.674462 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:56.677279 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.677953 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.677960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.677963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.678088 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:56.678599 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:56.678602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:56.678604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:56.678962 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:56.679209 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:56.679426 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 15:59:56.679703 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":404,"threads_started":1}
I20260924 15:59:56.681367 23488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:56.765841 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 23364
I20260924 15:59:56.782577 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 22999
I20260924 15:59:56.783912 23061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 15:59:56.889089 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 22999
2026-09-24T15:59:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4092 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:59:56 dist-test-slave-xn4h krb5kdc[23521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:59:57 dist-test-slave-xn4h krb5kdc[23521](info): set up 2 sockets
Sep 24 15:59:57 dist-test-slave-xn4h krb5kdc[23521](info): commencing operation
krb5kdc: starting...
W20260924 15:59:58.012706 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 15:59:58 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:59:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.190.keytab.
Entry for principal oryx/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.190.keytab.
I20260924 15:59:58.238253 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36701
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.190.keytab
--principal=oryx/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:58.569861 23534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:58.570161 23534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:58.570299 23534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:58.577693 23534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:59:58.577975 23534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:58.578132 23534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:58.578224 23534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:59:58.578334 23534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:59:58.588694 23534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.190.keytab
--principal=oryx/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:58.590108 23534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:58.591522 23534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:58.603685 23539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:58.603719 23542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:58.603749 23540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:58.604830 23534 server_base.cc:1034] running on GCE node
I20260924 15:59:58.605679 23534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:58.607278 23534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:58.608513 23534 hybrid_clock.cc:648] HybridClock initialized: now 1790265598608472 us; error 34 us; skew 500 ppm
I20260924 15:59:58.608877 23534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:58 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:58.624512 23534 init.cc:377] Logged in from keytab as oryx/127.16.233.190@KRBTEST.COM (short username oryx)
I20260924 15:59:58.626268 23534 webserver.cc:460] Webserver started at http://127.16.233.190:35679/ using document root <none> and password file <none>
I20260924 15:59:58.627106 23534 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:58.627264 23534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:58.627588 23534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:58.630827 23534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "4167211be21948888b84c382f610a8b4"
format_stamp: "Formatted at 2026-09-24 15:59:58 on dist-test-slave-xn4h"
server_key: "666011c4c23943540d23a16daf7a6e75"
server_key_iv: "fb4cd7c145b417fe12b4baef501a8214"
server_key_version: "encryptionkey@0"
I20260924 15:59:58.631652 23534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "4167211be21948888b84c382f610a8b4"
format_stamp: "Formatted at 2026-09-24 15:59:58 on dist-test-slave-xn4h"
server_key: "666011c4c23943540d23a16daf7a6e75"
server_key_iv: "fb4cd7c145b417fe12b4baef501a8214"
server_key_version: "encryptionkey@0"
I20260924 15:59:58.637441 23534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 15:59:58.641244 23549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:58.642692 23534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 15:59:58.642884 23534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "4167211be21948888b84c382f610a8b4"
format_stamp: "Formatted at 2026-09-24 15:59:58 on dist-test-slave-xn4h"
server_key: "666011c4c23943540d23a16daf7a6e75"
server_key_iv: "fb4cd7c145b417fe12b4baef501a8214"
server_key_version: "encryptionkey@0"
I20260924 15:59:58.643110 23534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:58.661201 23534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:58.665418 23534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:58.665722 23534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:58.684689 23534 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:36701
I20260924 15:59:58.684713 23601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:36701 every 8 connection(s)
I20260924 15:59:58.686205 23534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 15:59:58.689157 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 23534
I20260924 15:59:58.689365 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 15:59:58.690182 17318 external_mini_cluster.cc:1428] Setting key 4c4a3beee813697e27098b478550445f
I20260924 15:59:58.690641 23602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:59:58.703562 23602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: Bootstrap starting.
Sep 24 15:59:58 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 15:59:58.707108 23602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: Neither blocks nor log segments found. Creating new log.
I20260924 15:59:58.708328 23602 log.cc:824] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:59:58.711357 23602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: No bootstrap required, opened a new log
I20260924 15:59:58.713671 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:58.692395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.16.233.190:36701)
0924 15:59:58.693207 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:58.693218 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:58.693263 (+ 45us) server_negotiation.cc:407] Connection header received
0924 15:59:58.695505 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:58.695531 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:58.695965 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:58.696397 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:58.698012 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:58.699363 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:58.700643 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:58.701389 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:58.701642 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:58.705780 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:58.705788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:58.705803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:58.705847 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:58.709026 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:58.709782 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:58.709790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:58.709797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:58.709975 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:58.710419 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:58.710425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:58.710430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:58.711021 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:58.711362 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:58.711832 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 15:59:58.712323 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":381,"threads_started":1}
I20260924 15:59:58.719676 23602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } }
I20260924 15:59:58.720036 23602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:59:58.720110 23602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4167211be21948888b84c382f610a8b4, State: Initialized, Role: FOLLOWER
I20260924 15:59:58.720701 23602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } }
I20260924 15:59:58.720903 23602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:59:58.721078 23602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:59:58.721240 23602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:59:58.723613 23602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } }
I20260924 15:59:58.724150 23602 leader_election.cc:302] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4167211be21948888b84c382f610a8b4; no voters:
I20260924 15:59:58.724747 23602 leader_election.cc:288] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:59:58.724915 23607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:59:58.725615 23607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 1 LEADER]: Becoming Leader. State: Replica: 4167211be21948888b84c382f610a8b4, State: Running, Role: LEADER
I20260924 15:59:58.726181 23607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } }
I20260924 15:59:58.726999 23602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:59:58.729710 23608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } }
I20260924 15:59:58.729966 23609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4167211be21948888b84c382f610a8b4. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } }
I20260924 15:59:58.730196 23608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:58.730248 23609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:59:58.735656 23614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:59:58.740849 23614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:59:58.750766 23614 catalog_manager.cc:1347] Generated new cluster ID: 13b7d706a32a44ed856c10eab84e6939
I20260924 15:59:58.750910 23614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:59:58.768098 23614 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:59:58.769450 23614 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:59:58.778512 23614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: Generated new TSK 0
I20260924 15:59:58.779666 23614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab.
Entry for principal oryx/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab.
I20260924 15:59:58.833765 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36701
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab
--principal=oryx/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:59.175449 23628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:59.175735 23628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:59.175838 23628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:59.183207 23628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:59.183331 23628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:59.183444 23628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 15:59:59.193444 23628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab
--principal=oryx/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:59.194855 23628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:59.196195 23628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:59.209810 23633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:59.209861 23636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:59.209861 23634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:59.210891 23628 server_base.cc:1034] running on GCE node
I20260924 15:59:59.211546 23628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:59.212492 23628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:59.213752 23628 hybrid_clock.cc:648] HybridClock initialized: now 1790265599213704 us; error 37 us; skew 500 ppm
I20260924 15:59:59.214131 23628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:59 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:59.217449 23628 init.cc:377] Logged in from keytab as oryx/127.16.233.129@KRBTEST.COM (short username oryx)
I20260924 15:59:59.219065 23628 webserver.cc:460] Webserver started at http://127.16.233.129:40347/ using document root <none> and password file <none>
I20260924 15:59:59.219776 23628 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:59.219921 23628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:59.220263 23628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:59.223333 23628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "9f72561ef010464faf6a7ae00ab1ec7a"
format_stamp: "Formatted at 2026-09-24 15:59:59 on dist-test-slave-xn4h"
server_key: "521f4bcb26eeb2a8ac22e9adf4345122"
server_key_iv: "4f68437b4f3f06fb997ae82cf015d529"
server_key_version: "encryptionkey@0"
I20260924 15:59:59.224170 23628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "9f72561ef010464faf6a7ae00ab1ec7a"
format_stamp: "Formatted at 2026-09-24 15:59:59 on dist-test-slave-xn4h"
server_key: "521f4bcb26eeb2a8ac22e9adf4345122"
server_key_iv: "4f68437b4f3f06fb997ae82cf015d529"
server_key_version: "encryptionkey@0"
I20260924 15:59:59.229635 23628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 15:59:59.233474 23643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:59.235051 23628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 15:59:59.235244 23628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "9f72561ef010464faf6a7ae00ab1ec7a"
format_stamp: "Formatted at 2026-09-24 15:59:59 on dist-test-slave-xn4h"
server_key: "521f4bcb26eeb2a8ac22e9adf4345122"
server_key_iv: "4f68437b4f3f06fb997ae82cf015d529"
server_key_version: "encryptionkey@0"
I20260924 15:59:59.235466 23628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:59.248611 23628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:59.253322 23628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:59.253654 23628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:59.254819 23628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:59.256253 23628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:59.256357 23628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:59.256510 23628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:59.256582 23628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:59.298990 23628 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:39563
I20260924 15:59:59.299038 23756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:39563 every 8 connection(s)
I20260924 15:59:59.300408 23628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 15:59:59.302361 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 23628
I20260924 15:59:59.302582 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 15:59:59.303313 17318 external_mini_cluster.cc:1428] Setting key 783561e10cc498828608c387de1e7b08
Sep 24 15:59:59 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 15:59:59.324419 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:59.303249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:50687 (local address 127.16.233.190:36701)
0924 15:59:59.303790 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:59.303796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:59.304920 (+ 1124us) server_negotiation.cc:407] Connection header received
0924 15:59:59.307765 (+ 2845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:59.307771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:59.307850 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:59.307951 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:59.310231 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:59.311080 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:59.313452 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:59.314150 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:59.314261 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:59.318024 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:59.318030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:59.318033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:59.318072 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:59.320882 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:59.321846 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:59.321852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:59.321858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:59.321996 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:59.322521 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:59.322529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:59.322532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:59.322830 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:59.322992 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:59.323777 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 15:59:59.324040 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":282,"threads_started":1}
WARNING: no policy specified for oryx/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 15:59:59.325495 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:59.303835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.129:50687)
0924 15:59:59.304767 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:59.304813 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:59.307388 (+ 2575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:59.308270 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:59.308286 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:59.309092 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:59.309994 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:59.310024 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:59.311262 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:59.311271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:59.313280 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:59.313298 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:59.314442 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:59.314448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:59.314658 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:59.315401 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:59.315425 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:59.317748 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:59.321075 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:59.321086 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:59.321106 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:59.321711 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:59.322118 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:59.322123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:59.322128 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:59.322408 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:59.322970 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:59.322979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:59.323375 (+ 396us) client_negotiation.cc:769] Sending connection context
0924 15:59:59.323652 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 15:59:59.324236 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":386,"threads_started":1}
I20260924 15:59:59.327308 23757 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36701
I20260924 15:59:59.327647 23757 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.130@KRBTEST.COM" created.
I20260924 15:59:59.328581 23757 heartbeater.cc:511] Master 127.16.233.190:36701 requested a full tablet report, sending...
I20260924 15:59:59.331203 23566 ts_manager.cc:194] Registered new tserver with Master: 9f72561ef010464faf6a7ae00ab1ec7a (127.16.233.129:39563)
I20260924 15:59:59.333483 23566 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:50687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab.
Entry for principal oryx/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab.
I20260924 15:59:59.356006 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36701
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab
--principal=oryx/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:59:59.685575 23763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:59:59.685963 23763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:59:59.686123 23763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:59:59.697897 23763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:59:59.698057 23763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:59:59.698230 23763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 15:59:59.711897 23763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab
--principal=oryx/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 15:59:59.713261 23763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:59:59.714712 23763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:59:59.728225 23768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:59.728269 23769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:59:59.728302 23771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:59:59.728871 23763 server_base.cc:1034] running on GCE node
I20260924 15:59:59.729600 23763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:59:59.730792 23763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:59:59.732040 23763 hybrid_clock.cc:648] HybridClock initialized: now 1790265599731957 us; error 67 us; skew 500 ppm
I20260924 15:59:59.732384 23763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 15:59:59 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:59:59.735760 23763 init.cc:377] Logged in from keytab as oryx/127.16.233.130@KRBTEST.COM (short username oryx)
I20260924 15:59:59.737344 23763 webserver.cc:460] Webserver started at http://127.16.233.130:38903/ using document root <none> and password file <none>
I20260924 15:59:59.738057 23763 fs_manager.cc:359] Metadata directory not provided
I20260924 15:59:59.738207 23763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:59:59.738561 23763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:59:59.741677 23763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "7ef64d9eec314c388971bcb822442cdb"
format_stamp: "Formatted at 2026-09-24 15:59:59 on dist-test-slave-xn4h"
server_key: "c6239b428a31179ff7735f948f6c81f0"
server_key_iv: "379a90b195d1498368bcab8b1104c9d9"
server_key_version: "encryptionkey@0"
I20260924 15:59:59.742568 23763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "7ef64d9eec314c388971bcb822442cdb"
format_stamp: "Formatted at 2026-09-24 15:59:59 on dist-test-slave-xn4h"
server_key: "c6239b428a31179ff7735f948f6c81f0"
server_key_iv: "379a90b195d1498368bcab8b1104c9d9"
server_key_version: "encryptionkey@0"
I20260924 15:59:59.748059 23763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 15:59:59.751612 23778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:59.752938 23763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 15:59:59.753119 23763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "7ef64d9eec314c388971bcb822442cdb"
format_stamp: "Formatted at 2026-09-24 15:59:59 on dist-test-slave-xn4h"
server_key: "c6239b428a31179ff7735f948f6c81f0"
server_key_iv: "379a90b195d1498368bcab8b1104c9d9"
server_key_version: "encryptionkey@0"
I20260924 15:59:59.753330 23763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:59:59.774078 23763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:59:59.778805 23763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:59:59.779115 23763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:59:59.780110 23763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:59:59.781524 23763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:59:59.781629 23763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:59.781787 23763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:59:59.781862 23763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:59:59.814545 23763 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:46313
I20260924 15:59:59.814599 23890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:46313 every 8 connection(s)
I20260924 15:59:59.815845 23763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 15:59:59.817248 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 23763
I20260924 15:59:59.817447 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 15:59:59.817939 17318 external_mini_cluster.cc:1428] Setting key ec09b168a01b3db5dd5975bea546abda
Sep 24 15:59:59 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 15:59:59.838277 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:59.818637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:45735 (local address 127.16.233.190:36701)
0924 15:59:59.818850 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 15:59:59.818857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:59:59.820276 (+ 1419us) server_negotiation.cc:407] Connection header received
0924 15:59:59.822101 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:59:59.822106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:59:59.822193 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:59:59.822349 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:59:59.823844 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:59.824562 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:59.826156 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:59:59.826903 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:59:59.827013 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:59.831156 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:59:59.831164 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:59:59.831172 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:59:59.831218 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:59:59.834889 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:59.835635 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:59.835643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:59.835650 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:59.835814 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:59:59.836224 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:59:59.836232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:59:59.836238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:59:59.836628 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:59:59.836835 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 15:59:59.837511 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 15:59:59.837804 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 15:59:59.838929 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:59:59.819176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.130:45735)
0924 15:59:59.820104 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0924 15:59:59.820163 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 15:59:59.821833 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:59:59.822587 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:59:59.822595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:59:59.823093 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:59:59.823665 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:59.823683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:59.824716 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:59.824723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:59.825981 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:59:59.825997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:59:59.827350 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:59:59.827359 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:59:59.827586 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:59:59.828338 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:59:59.828365 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:59:59.830873 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:59:59.835043 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:59.835050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:59.835064 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:59.835480 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:59.835922 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:59:59.835925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:59:59.835928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:59:59.836102 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:59:59.836763 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:59:59.836769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:59:59.837116 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 15:59:59.837348 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 15:59:59.837768 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":378,"threads_started":1}
I20260924 15:59:59.840694 23891 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36701
WARNING: no policy specified for oryx/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 15:59:59.841008 23891 heartbeater.cc:464] Registering TS with master...
I20260924 15:59:59.841801 23891 heartbeater.cc:511] Master 127.16.233.190:36701 requested a full tablet report, sending...
I20260924 15:59:59.843492 23566 ts_manager.cc:194] Registered new tserver with Master: 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.131@KRBTEST.COM" created.
I20260924 15:59:59.844805 23566 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:45735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab.
Entry for principal oryx/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab.
I20260924 15:59:59.870064 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36701
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab
--principal=oryx/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:00.201323 23896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:00.201647 23896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:00.201817 23896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:00.209365 23896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:00.209520 23896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:00.209693 23896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:00:00.219815 23896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab
--principal=oryx/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:00.221268 23896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:00.222687 23896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:00.235845 23901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:00.235906 23902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:00.235931 23904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:00.236856 23896 server_base.cc:1034] running on GCE node
I20260924 16:00:00.237435 23896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:00.238407 23896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:00.239612 23896 hybrid_clock.cc:648] HybridClock initialized: now 1790265600239553 us; error 59 us; skew 500 ppm
I20260924 16:00:00.239952 23896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:00 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265600, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:00.243101 23896 init.cc:377] Logged in from keytab as oryx/127.16.233.131@KRBTEST.COM (short username oryx)
I20260924 16:00:00.244596 23896 webserver.cc:460] Webserver started at http://127.16.233.131:43561/ using document root <none> and password file <none>
I20260924 16:00:00.245301 23896 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:00.245429 23896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:00.245743 23896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:00.248941 23896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "ec6e7c31a45049689e8e72ade7a08763"
format_stamp: "Formatted at 2026-09-24 16:00:00 on dist-test-slave-xn4h"
server_key: "e74f8e823a94dc45ee5daf6549f4cf2a"
server_key_iv: "3475d4e0d2d378a08db95d2c2ffeb582"
server_key_version: "encryptionkey@0"
I20260924 16:00:00.249778 23896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "ec6e7c31a45049689e8e72ade7a08763"
format_stamp: "Formatted at 2026-09-24 16:00:00 on dist-test-slave-xn4h"
server_key: "e74f8e823a94dc45ee5daf6549f4cf2a"
server_key_iv: "3475d4e0d2d378a08db95d2c2ffeb582"
server_key_version: "encryptionkey@0"
I20260924 16:00:00.255051 23896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:00:00.258643 23911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:00.259977 23896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:00:00.260150 23896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "ec6e7c31a45049689e8e72ade7a08763"
format_stamp: "Formatted at 2026-09-24 16:00:00 on dist-test-slave-xn4h"
server_key: "e74f8e823a94dc45ee5daf6549f4cf2a"
server_key_iv: "3475d4e0d2d378a08db95d2c2ffeb582"
server_key_version: "encryptionkey@0"
I20260924 16:00:00.260346 23896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:00.289724 23896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:00.294003 23896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:00.294363 23896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:00.295425 23896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:00.296877 23896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:00.296950 23896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:00.297099 23896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:00.297158 23896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:00.331804 23896 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:45151
I20260924 16:00:00.331848 24025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:45151 every 8 connection(s)
I20260924 16:00:00.333214 23896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:00:00.336843 23757 heartbeater.cc:503] Master 127.16.233.190:36701 was elected leader, sending a full tablet report...
I20260924 16:00:00.341570 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 23896
I20260924 16:00:00.341758 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:00:00.342367 17318 external_mini_cluster.cc:1428] Setting key cd65a4a810bef66fc477854f63dee500
Sep 24 16:00:00 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265600, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:00.354354 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.335850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:40803 (local address 127.16.233.190:36701)
0924 16:00:00.336057 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:00.336064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:00.337406 (+ 1342us) server_negotiation.cc:407] Connection header received
0924 16:00:00.339556 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:00.339564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:00.339678 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:00.339765 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:00.341438 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.342658 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.344317 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.344740 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.344874 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.348507 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:00.348521 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:00.348526 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:00.348559 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:00.351144 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.351933 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.351939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.351943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.352082 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.352536 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.352541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.352544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.352837 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:00.353002 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:00.353731 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 16:00:00.353968 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:00:00.355319 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.336398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.131:40803)
0924 16:00:00.337241 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:00.337297 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:00.339242 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:00.340031 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:00.340040 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:00.340647 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:00.341253 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.341272 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.342816 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.342820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.344133 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.344147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.344981 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.344988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.345236 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.345879 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:00.345903 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:00.348277 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:00.351320 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.351330 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.351351 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.351802 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.352199 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.352204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.352215 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.352416 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.352976 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:00.352984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:00.353354 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 16:00:00.353623 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 16:00:00.354184 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":375,"threads_started":1}
I20260924 16:00:00.356980 24026 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36701
I20260924 16:00:00.357367 24026 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:00.358274 24026 heartbeater.cc:511] Master 127.16.233.190:36701 requested a full tablet report, sending...
I20260924 16:00:00.359750 23565 ts_manager.cc:194] Registered new tserver with Master: ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151)
I20260924 16:00:00.360641 23565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:40803
I20260924 16:00:00.370605 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.233.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.189.keytab.
Entry for principal oryx/127.16.233.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.189.keytab.
I20260924 16:00:00.419938 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.189:41823
--webserver_interface=127.16.233.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.189.keytab
--principal=oryx/127.16.233.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36701,127.16.233.189:41823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:00.756500 24031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:00.756866 24031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:00.757020 24031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:00.764725 24031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:00:00.764881 24031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:00.764995 24031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:00.765097 24031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:00:00.765179 24031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:00:00.775343 24031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36701,127.16.233.189:41823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.189.keytab
--principal=oryx/127.16.233.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.189:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.233.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
W20260924 16:00:00.775686 24031 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.233.190:36701,127.16.233.189:41823'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 16:00:00.776705 24031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:00.778002 24031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:00.789436 24037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:00.789389 24036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:00.790668 24039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:00.791885 24031 server_base.cc:1034] running on GCE node
I20260924 16:00:00.792793 24031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:00.794363 24031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:00.795644 24031 hybrid_clock.cc:648] HybridClock initialized: now 1790265600795550 us; error 69 us; skew 500 ppm
I20260924 16:00:00.796025 24031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:00 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265600, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:00.799433 24031 init.cc:377] Logged in from keytab as oryx/127.16.233.189@KRBTEST.COM (short username oryx)
I20260924 16:00:00.801203 24031 webserver.cc:460] Webserver started at http://127.16.233.189:37773/ using document root <none> and password file <none>
I20260924 16:00:00.801940 24031 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:00.802037 24031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:00.802475 24031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:00.805565 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data/instance:
uuid: "85ffe92f9a0e479c940899f0a207cec3"
format_stamp: "Formatted at 2026-09-24 16:00:00 on dist-test-slave-xn4h"
server_key: "3abec81de02c513e286100355cd6695e"
server_key_iv: "b0908b4e24c1c9954915f986d2d86beb"
server_key_version: "encryptionkey@0"
I20260924 16:00:00.806360 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal/instance:
uuid: "85ffe92f9a0e479c940899f0a207cec3"
format_stamp: "Formatted at 2026-09-24 16:00:00 on dist-test-slave-xn4h"
server_key: "3abec81de02c513e286100355cd6695e"
server_key_iv: "b0908b4e24c1c9954915f986d2d86beb"
server_key_version: "encryptionkey@0"
I20260924 16:00:00.811794 24031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:00:00.815570 24046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:00.817054 24031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:00:00.817260 24031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal
uuid: "85ffe92f9a0e479c940899f0a207cec3"
format_stamp: "Formatted at 2026-09-24 16:00:00 on dist-test-slave-xn4h"
server_key: "3abec81de02c513e286100355cd6695e"
server_key_iv: "b0908b4e24c1c9954915f986d2d86beb"
server_key_version: "encryptionkey@0"
I20260924 16:00:00.817489 24031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:00.834582 24031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:00.839228 24031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:00.839507 24031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:00.847609 23891 heartbeater.cc:503] Master 127.16.233.190:36701 was elected leader, sending a full tablet report...
I20260924 16:00:00.859061 24031 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.189:41823
I20260924 16:00:00.859081 24098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.189:41823 every 8 connection(s)
I20260924 16:00:00.860535 24031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
I20260924 16:00:00.862005 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 24031
I20260924 16:00:00.865339 24099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:00.872195 24099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:00:00 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.189@KRBTEST.COM
Sep 24 16:00:00 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265600, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.189@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:00.886682 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.864405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.16.233.189:41823)
0924 16:00:00.865232 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:00.865247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:00.865300 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:00:00.867046 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:00.867067 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:00.867429 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:00.867838 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:00.869333 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.870806 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.872056 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.872792 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.872992 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.878244 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:00.878252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:00.878263 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:00.878380 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:00.881480 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.882435 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.882442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.882449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.882628 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.883057 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.883063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.883066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.883857 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:00.884138 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:00.884713 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 16:00:00.885179 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"spinlock_wait_cycles":15104,"thread_start_us":337,"threads_started":1}
I20260924 16:00:00.890978 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.874882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.16.233.190:36701)
0924 16:00:00.876177 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:00.876182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:00.876196 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:00:00.876252 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:00.876255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:00.876317 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:00.876437 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:00.878073 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.879032 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.880642 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.881057 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.881208 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.884906 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:00.884913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:00.884918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:00.884960 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:00.887918 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.888691 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.888697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.888703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.888835 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.889290 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.889296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.889301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.889707 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:00.889867 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:00.890367 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 16:00:00.890601 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":1095,"threads_started":1}
I20260924 16:00:00.891021 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.874838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54098)
0924 16:00:00.875589 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:00.875642 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:00.875797 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:00.876499 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:00.876506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:00.877520 (+ 1014us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:00.877910 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.877920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.879182 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.879188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.880463 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.880477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.881284 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.881291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.881483 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.882094 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:00.882111 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:00.884677 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:00.888063 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.888077 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.888091 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.888551 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.888950 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.888960 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.888968 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.889106 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.889840 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:00.889847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:00.890026 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 16:00:00.890230 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 16:00:00.890595 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":379,"threads_started":1}
I20260924 16:00:00.892779 24099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:00:00 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265600, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.189@KRBTEST.COM for oryx/127.16.233.189@KRBTEST.COM
I20260924 16:00:00.904997 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.893417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.0.0.1:51578)
0924 16:00:00.893631 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:00.893655 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:00.893760 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:00.894050 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:00.894054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:00.894452 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:00.894584 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.894591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.895512 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.895515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.896383 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.896394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.897014 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.897019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.897157 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.897735 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:00.897751 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:00.899763 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:00.902862 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.902866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.902869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.903292 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.903657 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.903668 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.903671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.903808 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.904275 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:00.904278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:00.904407 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 16:00:00.904520 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 16:00:00.904741 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 16:00:00.905238 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.893674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.16.233.189:41823)
0924 16:00:00.893830 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:00.893835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:00.893850 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:00:00.893889 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:00.893892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:00.893941 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:00.894016 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:00.894698 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.895391 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.896501 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.896882 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.896970 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.899929 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:00.899934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:00.899937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:00.899988 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:00.902722 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.903409 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.903413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.903415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.903533 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.903908 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.903911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.903913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.904167 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:00.904276 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:00.904610 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 16:00:00.904884 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":56}
I20260924 16:00:00.910019 24099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Bootstrap starting.
I20260924 16:00:00.913256 24099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:00.914458 24099 log.cc:824] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:00.917368 24099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: No bootstrap required, opened a new log
I20260924 16:00:00.924710 24099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } }
I20260924 16:00:00.925082 24099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:00.925182 24099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ffe92f9a0e479c940899f0a207cec3, State: Initialized, Role: FOLLOWER
I20260924 16:00:00.925804 24099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } }
I20260924 16:00:00.926815 24106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } } }
I20260924 16:00:00.927163 24106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:00:00.927977 24099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:00:00.938122 24117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:00:00.938555 24117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:00:00.941635 24031 master_runner.cc:428] Detected that this master 85ffe92f9a0e479c940899f0a207cec3 is joining an existing cluster
I20260924 16:00:00.941751 24031 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.233.189:41823
I20260924 16:00:00.967526 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.949793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.16.233.190:36701)
0924 16:00:00.950000 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:00.950006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:00.950597 (+ 591us) server_negotiation.cc:407] Connection header received
0924 16:00:00.950931 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:00.950937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:00.951053 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:00.951131 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:00.952892 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.953607 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.955952 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.956742 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.956868 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.959900 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:00.959909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:00.959916 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:00.959964 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:00.963853 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.964847 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.964857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.964863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.965032 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.965547 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.965555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.965561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.965995 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:00.966142 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:00.966850 (+ 708us) server_negotiation.cc:299] Negotiation successful
0924 16:00:00.967135 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:00:00.967693 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.950565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.16.233.189:41823)
0924 16:00:00.950713 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:00.950718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:00.951383 (+ 665us) server_negotiation.cc:407] Connection header received
0924 16:00:00.951609 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:00.951614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:00.951704 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:00.951784 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:00.953252 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.954409 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.956346 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.957057 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.957152 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.960166 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:00.960174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:00.960179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:00.960213 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:00.964540 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.965359 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.965364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.965367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.965521 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.965974 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.965979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.965982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.966444 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:00.966594 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:00.967079 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 16:00:00.967300 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":35}
I20260924 16:00:00.968766 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.949793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54100)
0924 16:00:00.950487 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:00.950507 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:00.950740 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:00.951380 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:00.951388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:00.952267 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:00.952737 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.952748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.953737 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.953745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.955785 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.955802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.957228 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.957237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.957465 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.958223 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:00.958259 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:00.959668 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:00.964036 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.964044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.964051 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.964696 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.965202 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.965208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.965214 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.965399 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.966232 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:00.966238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:00.966517 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 16:00:00.967826 (+ 1309us) client_negotiation.cc:240] Negotiation successful
0924 16:00:00.968177 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":3,"thread_start_us":1011,"threads_started":1}
I20260924 16:00:00.968901 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.950564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.0.0.1:51582)
0924 16:00:00.951282 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:00.951301 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:00.951468 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:00.951983 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:00.951987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:00.952758 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:00.953088 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.953101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.954627 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.954632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.956201 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.956216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.957514 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.957521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.957723 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.958739 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:00.958776 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:00.959950 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:00.964704 (+ 4754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.964708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.964714 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.965228 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.965688 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.965692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.965696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.965849 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.966670 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:00.966675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:00.966816 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 16:00:00.967945 (+ 1129us) client_negotiation.cc:240] Negotiation successful
0924 16:00:00.968390 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":137,"thread_start_us":497,"threads_started":1}
WARNING: no policy specified for oryx/127.16.233.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.188@KRBTEST.COM" created.
I20260924 16:00:00.973008 24106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:00.973366 24106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } }
I20260924 16:00:00.981606 24106 leader_election.cc:288] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4167211be21948888b84c382f610a8b4 (127.16.233.190:36701)
I20260924 16:00:00.982669 23576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85ffe92f9a0e479c940899f0a207cec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4167211be21948888b84c382f610a8b4" is_pre_election: true
I20260924 16:00:00.984001 24047 leader_election.cc:302] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 85ffe92f9a0e479c940899f0a207cec3; no voters: 4167211be21948888b84c382f610a8b4
I20260924 16:00:00.984771 24106 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 24 16:00:00 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.190@KRBTEST.COM for oryx/127.16.233.189@KRBTEST.COM
I20260924 16:00:00.997503 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.976778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.0.0.1:51590)
0924 16:00:00.977482 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:00.977533 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:00.977785 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:00.978578 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:00.978588 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:00.979528 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:00.979876 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.979898 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.981488 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.981494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.983522 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:00.983545 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.985116 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.985122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:00.985339 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.986152 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:00.986178 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:00.989436 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:00.993958 (+ 4522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.993986 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.993996 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.994792 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.995322 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:00.995328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:00.995331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:00.995520 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:00.996322 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:00.996330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:00.996596 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 16:00:00.996769 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 16:00:00.997080 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":516,"threads_started":1}
I20260924 16:00:00.997610 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:00.977096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.16.233.189:41823)
0924 16:00:00.977294 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:00.977302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:00.977623 (+ 321us) server_negotiation.cc:407] Connection header received
0924 16:00:00.978017 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:00.978025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:00.978150 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:00.978227 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:00.980066 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.981304 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.983737 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:00.984606 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:00.984736 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:00.989710 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:00.989723 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:00.989729 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:00.989775 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:00.993735 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.994973 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.994979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.994984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.995171 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:00.995684 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:00.995690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:00.995693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:00.996161 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:00.996338 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:00.996963 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 16:00:00.997231 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:00:00.999954 23565 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.233.189:41823: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3"
member_type: NON_VOTER
last_known_addr {
host: "127.16.233.189"
port: 41823
}
attrs {
promote: true
}
}
dest_uuid: "4167211be21948888b84c382f610a8b4"
cas_config_opid_index: -1
I20260924 16:00:01.001116 23565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.188.keytab.
Entry for principal oryx/127.16.233.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.188.keytab.
I20260924 16:00:01.004884 24130 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 85ffe92f9a0e479c940899f0a207cec3 (127.16.233.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } }
I20260924 16:00:01.006515 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.188:46427
--webserver_interface=127.16.233.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.188.keytab
--principal=oryx/127.16.233.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36701,127.16.233.189:41823,127.16.233.188:46427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:00:01.006871 24074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:01.012156 24074 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Refusing update from remote peer 4167211be21948888b84c382f610a8b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 16:00:01.014914 24129 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:01.015053 24130 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.233.189:41823
I20260924 16:00:01.016932 24031 master.cc:537] Master@127.16.233.189:41823 shutting down...
I20260924 16:00:01.018080 24134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } }
I20260924 16:00:01.018586 24134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
W20260924 16:00:01.018880 23553 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 16:00:01.030915 23553 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 -> Peer 85ffe92f9a0e479c940899f0a207cec3 (127.16.233.189:41823): Couldn't send request to peer 85ffe92f9a0e479c940899f0a207cec3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 16:00:01.033023 24031 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:01.033480 24031 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:01.033696 24031 tablet_replica.cc:330] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: stopping tablet replica
I20260924 16:00:01.050812 24031 master.cc:552] Master@127.16.233.189:41823 shutdown complete.
I20260924 16:00:01.050930 24031 master_runner.cc:305] Clearing existing system tablet
I20260924 16:00:01.051344 24031 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:01.051462 24031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:01.053824 24031 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:00:01.055454 24138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:01.055811 24031 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:01.056115 24031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 16:00:01.056218 24031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal
uuid: "85ffe92f9a0e479c940899f0a207cec3"
format_stamp: "Formatted at 2026-09-24 16:00:00 on dist-test-slave-xn4h"
server_key: "3abec81de02c513e286100355cd6695e"
server_key_iv: "b0908b4e24c1c9954915f986d2d86beb"
server_key_version: "encryptionkey@0"
I20260924 16:00:01.060263 24031 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:01.065740 24031 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:00:01.065979 24031 log.cc:1196] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 16:00:01.066454 24031 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Deleting consensus metadata
I20260924 16:00:01.066728 24031 master_runner.cc:315] Copying system tablet from 127.16.233.190:36701
I20260924 16:00:01.071148 24031 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.233.190:36701
I20260924 16:00:01.084643 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.072006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.16.233.190:36701)
0924 16:00:01.072271 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.072280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.072821 (+ 541us) server_negotiation.cc:407] Connection header received
0924 16:00:01.073047 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.073053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.073146 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.073267 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.074101 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.074945 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.076263 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.076721 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.076866 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.078939 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.078944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.078948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.078983 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.081745 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.082562 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.082569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.082574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.082706 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.083095 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.083102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.083106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.083402 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.083559 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.083988 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.084233 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:00:01.084698 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.071951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54110)
0924 16:00:01.072692 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.072708 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.072905 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.073288 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.073292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.073725 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:01.073943 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.073953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.075120 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.075126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.076082 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.076095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.076887 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.076893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.077100 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.077793 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:01.077811 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:01.078725 (+ 914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:01.081912 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.081917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.081922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.082430 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.082825 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.082830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.082834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.082985 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.083522 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:01.083527 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:01.083706 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.083871 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.084233 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":493,"threads_started":1}
I20260924 16:00:01.085984 23587 tablet_copy_service.cc:140] P 4167211be21948888b84c382f610a8b4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 85ffe92f9a0e479c940899f0a207cec3 ({username='oryx', principal='oryx/127.16.233.189@KRBTEST.COM'} at 127.0.0.1:54110)
I20260924 16:00:01.086243 23587 tablet_copy_service.cc:161] P 4167211be21948888b84c382f610a8b4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 85ffe92f9a0e479c940899f0a207cec3 at {username='oryx', principal='oryx/127.16.233.189@KRBTEST.COM'} at 127.0.0.1:54110: session id = 85ffe92f9a0e479c940899f0a207cec3-00000000000000000000000000000000
I20260924 16:00:01.089479 23587 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:00:01.091732 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:00:01.096668 24031 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: tablet copy: Starting download of 0 data blocks...
I20260924 16:00:01.097028 24031 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: tablet copy: Starting download of 1 WAL segments...
I20260924 16:00:01.099821 24031 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:00:01.103577 23587 tablet_copy_service.cc:342] P 4167211be21948888b84c382f610a8b4: Request end of tablet copy session 85ffe92f9a0e479c940899f0a207cec3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.233.189@KRBTEST.COM'} at 127.0.0.1:54110
I20260924 16:00:01.103783 23587 tablet_copy_service.cc:433] P 4167211be21948888b84c382f610a8b4: ending tablet copy session 85ffe92f9a0e479c940899f0a207cec3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 85ffe92f9a0e479c940899f0a207cec3
I20260924 16:00:01.114107 24031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:01.116927 24145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:01.122089 24146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:01.129437 24148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:01.130421 24031 server_base.cc:1034] running on GCE node
I20260924 16:00:01.130939 24031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:01.131651 24031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:01.132859 24031 hybrid_clock.cc:648] HybridClock initialized: now 1790265601132796 us; error 39 us; skew 500 ppm
I20260924 16:00:01.133112 24031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:01 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:01.135161 24031 init.cc:377] Logged in from keytab as oryx/127.16.233.189@KRBTEST.COM (short username oryx)
I20260924 16:00:01.136613 24031 webserver.cc:460] Webserver started at http://127.16.233.189:44423/ using document root <none> and password file <none>
I20260924 16:00:01.136981 24031 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:01.137080 24031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:01.139868 24031 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:00:01.141883 24155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:01.142369 24031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 16:00:01.142534 24031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal
uuid: "85ffe92f9a0e479c940899f0a207cec3"
format_stamp: "Formatted at 2026-09-24 16:00:00 on dist-test-slave-xn4h"
server_key: "3abec81de02c513e286100355cd6695e"
server_key_iv: "b0908b4e24c1c9954915f986d2d86beb"
server_key_version: "encryptionkey@0"
I20260924 16:00:01.142735 24031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:01.168136 24031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:01.171994 24031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:01.199069 24031 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.189:41823
I20260924 16:00:01.199090 24206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.189:41823 every 8 connection(s)
I20260924 16:00:01.200139 24031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
I20260924 16:00:01.207427 24207 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:00:01.210150 24207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Bootstrap starting.
I20260924 16:00:01.225270 24207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:00:01.225843 24207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Bootstrap complete.
I20260924 16:00:01.226805 24207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } }
I20260924 16:00:01.227082 24207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85ffe92f9a0e479c940899f0a207cec3, State: Initialized, Role: LEARNER
I20260924 16:00:01.227314 24207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } }
I20260924 16:00:01.228303 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } }
I20260924 16:00:01.228569 24207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:00:01.228735 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:00:01.238183 24221 catalog_manager.cc:1259] Loaded cluster ID: 13b7d706a32a44ed856c10eab84e6939
I20260924 16:00:01.238348 24221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: loading cluster ID for follower catalog manager: success
I20260924 16:00:01.241233 24221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: acquiring CA information for follower catalog manager: success
I20260924 16:00:01.243449 24221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:00:01.363488 24026 heartbeater.cc:503] Master 127.16.233.190:36701 was elected leader, sending a full tablet report...
W20260924 16:00:01.477159 24133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:01.477589 24133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:01.477746 24133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:01.485152 24133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:00:01.485302 24133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:01.485410 24133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:01.485515 24133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:00:01.485592 24133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:00:01.495762 24133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36701,127.16.233.189:41823,127.16.233.188:46427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.188.keytab
--principal=oryx/127.16.233.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.188:46427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.233.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:01.497129 24133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:01.498603 24133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:01.511228 24229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:01.511348 24227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:01.511189 24226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:01.513108 24133 server_base.cc:1034] running on GCE node
I20260924 16:00:01.514024 24133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:01.515573 24133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:01.516812 24133 hybrid_clock.cc:648] HybridClock initialized: now 1790265601516754 us; error 53 us; skew 500 ppm
I20260924 16:00:01.517205 24133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:01 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:01.520401 24133 init.cc:377] Logged in from keytab as oryx/127.16.233.188@KRBTEST.COM (short username oryx)
I20260924 16:00:01.521941 24133 webserver.cc:460] Webserver started at http://127.16.233.188:39575/ using document root <none> and password file <none>
I20260924 16:00:01.522782 24133 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:01.522931 24133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:01.523247 24133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:01.526289 24133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data/instance:
uuid: "1c5044a027d640be8f94f8f5bc5ee50a"
format_stamp: "Formatted at 2026-09-24 16:00:01 on dist-test-slave-xn4h"
server_key: "56604e1ad3f5127f488ff6e6343afc6b"
server_key_iv: "c1a529d77e54e81607644788b2584ea9"
server_key_version: "encryptionkey@0"
I20260924 16:00:01.527202 24133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal/instance:
uuid: "1c5044a027d640be8f94f8f5bc5ee50a"
format_stamp: "Formatted at 2026-09-24 16:00:01 on dist-test-slave-xn4h"
server_key: "56604e1ad3f5127f488ff6e6343afc6b"
server_key_iv: "c1a529d77e54e81607644788b2584ea9"
server_key_version: "encryptionkey@0"
I20260924 16:00:01.533262 24133 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260924 16:00:01.536958 24236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:01.538506 24133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:00:01.538689 24133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal
uuid: "1c5044a027d640be8f94f8f5bc5ee50a"
format_stamp: "Formatted at 2026-09-24 16:00:01 on dist-test-slave-xn4h"
server_key: "56604e1ad3f5127f488ff6e6343afc6b"
server_key_iv: "c1a529d77e54e81607644788b2584ea9"
server_key_version: "encryptionkey@0"
I20260924 16:00:01.538901 24133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:01.557086 24133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:01.561817 24133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:01.562273 24133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:01.576820 24245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.568290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.16.233.189:41823)
0924 16:00:01.569116 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.569121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.569141 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:00:01.569202 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.569205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.569273 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.569404 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:01.569846 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.571490 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.574264 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.575723 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.575932 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.576116 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.576341 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.576471 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":459,"threads_started":1}
I20260924 16:00:01.576817 24244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.567952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.0.0.1:51598)
0924 16:00:01.568401 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.568418 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.568612 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.569449 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.569456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.569486 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:01.569691 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.569700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.571684 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.571695 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.574037 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.574062 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.575951 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.575957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.576121 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.576201 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.576316 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.576465 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":247,"threads_started":1}
I20260924 16:00:01.578322 24182 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 16:00:01.578954 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4167211be21948888b84c382f610a8b4. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } }
I20260924 16:00:01.579232 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:00:01.590672 24133 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.188:46427
I20260924 16:00:01.590718 24291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.188:46427 every 8 connection(s)
I20260924 16:00:01.592093 24133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
I20260924 16:00:01.595856 24292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:01.599903 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 24133
I20260924 16:00:01.604202 24292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:00:01 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.188@KRBTEST.COM
Sep 24 16:00:01 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.188@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:01.626022 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.608206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.16.233.190:36701)
0924 16:00:01.608739 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.608747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.609237 (+ 490us) server_negotiation.cc:407] Connection header received
0924 16:00:01.609462 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.609467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.609537 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.609665 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.611617 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.612564 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.614274 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.615100 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.615271 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.619564 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.619569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.619572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.619607 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.623052 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.623900 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.623904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.623906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.624032 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.624508 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.624513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.624516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.624906 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.625051 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.625539 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.625726 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":276,"threads_started":1}
I20260924 16:00:01.627187 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.602376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.16.233.188:46427)
0924 16:00:01.603261 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.603275 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.603320 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:00:01.605151 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.605183 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.605643 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.606058 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.607922 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.609893 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.611932 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.612597 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.612823 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.618660 (+ 5837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.618672 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.618690 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.618751 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.621958 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.622893 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.622901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.622908 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.623072 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.623528 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.623536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.623542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.624314 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.624612 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.625084 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.625589 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":405,"threads_started":1}
I20260924 16:00:01.627220 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.608192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54126)
0924 16:00:01.609086 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.609121 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.609324 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.609728 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.609735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.611121 (+ 1386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:01.611446 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.611457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.612715 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.612719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.614101 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.614115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.615367 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.615373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.615507 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.616380 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:01.616405 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:01.619302 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:01.623181 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.623195 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.623204 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.623778 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.624146 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.624151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.624154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.624402 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.625037 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:01.625045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:01.625253 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.625474 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.625763 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"spinlock_wait_cycles":22912,"thread_start_us":504,"threads_started":1}
I20260924 16:00:01.628916 24292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:00:01 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.188@KRBTEST.COM for oryx/127.16.233.189@KRBTEST.COM
I20260924 16:00:01.644332 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.629799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.0.0.1:51602)
0924 16:00:01.630208 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.630226 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.630372 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.630708 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.630711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.631284 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:01.631579 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.631593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.632949 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.632953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.633893 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.633906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.634699 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.634705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.634838 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.635468 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:01.635486 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:01.637568 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:01.641513 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.641516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.641520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.641922 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.642420 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.642424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.642426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.642569 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.643332 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:01.643337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:01.643516 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.643676 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.643967 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260924 16:00:01.644562 24245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.629749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.16.233.189:41823)
0924 16:00:01.629936 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.629942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.630325 (+ 383us) server_negotiation.cc:407] Connection header received
0924 16:00:01.630491 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.630495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.630564 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.630655 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.631746 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.632803 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.634032 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.634550 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.634665 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.637764 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.637768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.637772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.637805 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.641208 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.642041 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.642045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.642048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.642247 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.642705 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.642708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.642710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.643139 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.643266 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.643755 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.643980 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:00:01.645608 24292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:00:01 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.188@KRBTEST.COM for oryx/127.16.233.188@KRBTEST.COM
I20260924 16:00:01.659389 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.646283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:46427 (local address 127.0.0.1:40010)
0924 16:00:01.646506 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.646525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.646619 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.646950 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.646953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.647515 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:01.647749 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.647760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.648696 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.648699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.649539 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.649550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.650182 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.650185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.650410 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.651020 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:01.651036 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:01.653087 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:01.656410 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.656415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.656419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.656862 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.657301 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.657306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.657309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.657492 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.658240 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:01.658244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:01.658427 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.658581 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.658910 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 16:00:01.659559 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.646517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.16.233.188:46427)
0924 16:00:01.646676 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.646680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.646694 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:00:01.646731 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.646734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.646823 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.646915 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.647872 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.648589 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.649674 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.650059 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.650158 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.653278 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.653283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.653286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.653320 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.656239 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.656986 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.656992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.656995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.657190 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.657599 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.657606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.657609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.658125 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.658281 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.658707 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.659092 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":49}
I20260924 16:00:01.664834 24292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Bootstrap starting.
I20260924 16:00:01.668144 24292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:01.669200 24292 log.cc:824] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:01.672323 24292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: No bootstrap required, opened a new log
I20260924 16:00:01.680063 24292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } }
I20260924 16:00:01.680461 24292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:01.680608 24292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5044a027d640be8f94f8f5bc5ee50a, State: Initialized, Role: FOLLOWER
I20260924 16:00:01.681272 24292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } }
I20260924 16:00:01.682351 24299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } } }
I20260924 16:00:01.682778 24299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:00:01.683576 24292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:00:01.695884 24310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:00:01.696153 24310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:00:01.699529 24133 master_runner.cc:428] Detected that this master 1c5044a027d640be8f94f8f5bc5ee50a is joining an existing cluster
I20260924 16:00:01.699750 24133 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.233.188:46427
I20260924 16:00:01.730415 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.705095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.16.233.190:36701)
0924 16:00:01.705309 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.705315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.707459 (+ 2144us) server_negotiation.cc:407] Connection header received
0924 16:00:01.707838 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.707848 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.708015 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.708110 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.711378 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.712916 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.715897 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.716878 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.717036 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.722009 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.722015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.722020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.722084 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.725919 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.727097 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.727105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.727109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.727275 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.727789 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.727796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.727802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.728225 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.728414 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.729050 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.729361 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:00:01.731992 24316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.704285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54142)
0924 16:00:01.707317 (+ 3032us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.707343 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.707641 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.708461 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.708470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.710244 (+ 1774us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:01.711116 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.711136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.713237 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.713246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.715619 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.715645 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.717773 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.717783 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.718138 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.719674 (+ 1536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:01.719705 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:01.721750 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:01.726082 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.726087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.726091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.726658 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.727427 (+ 769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.727432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.727436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.727652 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.728492 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:01.728498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:01.728723 (+ 225us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.730842 (+ 2119us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.731348 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2774,"thread_start_us":2373,"threads_started":1}
I20260924 16:00:01.735563 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 23628
I20260924 16:00:01.737963 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.705619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.16.233.188:46427)
0924 16:00:01.705799 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.705804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.719338 (+ 13534us) server_negotiation.cc:407] Connection header received
0924 16:00:01.719531 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.719541 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.719629 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.719768 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.721622 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.722810 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.725095 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.725829 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.725931 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.728898 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.728907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.728916 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.728950 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.733879 (+ 4929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.735043 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.735050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.735055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.735222 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.736036 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.736042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.736045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.736514 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.736691 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.737208 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.737543 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:00:01.739005 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.705474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:46427 (local address 127.0.0.1:40022)
0924 16:00:01.719178 (+ 13704us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.719205 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.719386 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.719943 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.719948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.720727 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:01.721449 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.721464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.723212 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.723218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.724935 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.724953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.726249 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.726256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.726452 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.727093 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:01.727118 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:01.728693 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:01.734379 (+ 5686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.734384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.734388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.734918 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.735389 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.735397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.735400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.735563 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.736762 (+ 1199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:01.736767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:01.736955 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.738198 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.738545 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13455,"thread_start_us":15533,"threads_started":1}
I20260924 16:00:01.740754 24245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.707612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.16.233.189:41823)
0924 16:00:01.707794 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.707800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.721705 (+ 13905us) server_negotiation.cc:407] Connection header received
0924 16:00:01.721966 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.721972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.722054 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.722128 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.724415 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.725527 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.728210 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.728901 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.728995 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.733319 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.733326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.733332 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.733384 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.736978 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.737821 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.737826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.737829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.738009 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.738507 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.738513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.738516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.738857 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.738999 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.739998 (+ 999us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.740268 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:00:01.742130 24317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.708175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.0.0.1:51612)
0924 16:00:01.721587 (+ 13412us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.721607 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.721807 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.722382 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.722388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.723769 (+ 1381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:01.724168 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.724187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.725730 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.725736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.728044 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.728059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.729337 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.729345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.729516 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.730441 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:01.730466 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:01.733091 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:01.737133 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.737137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.737141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.737669 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.738167 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.738171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.738175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.738369 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.739087 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:01.739091 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:01.739354 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.740985 (+ 1631us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.741446 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13275,"thread_start_us":10477,"threads_started":1}
Sep 24 16:00:01 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.190@KRBTEST.COM for oryx/127.16.233.188@KRBTEST.COM
I20260924 16:00:01.763847 24244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.744252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:46427 (local address 127.0.0.1:40024)
0924 16:00:01.744532 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.744558 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.744689 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.745097 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.745104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.745708 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:01.745986 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.746004 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.747700 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.747708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.749694 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.749718 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.750891 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.750900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.751176 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.752033 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:01.752069 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:01.755402 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:01.760651 (+ 5249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.760666 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.760670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.761331 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.761827 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.761833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.761836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.762023 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.762722 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:01.762727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:01.762905 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.763070 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.763450 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 16:00:01.763840 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.744616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.16.233.188:46427)
0924 16:00:01.744771 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.744779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.744794 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:00:01.744887 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.744890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.744950 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.745067 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.746223 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.747491 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.749888 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.750665 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.750859 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.755671 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.755678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.755682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.755727 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.760480 (+ 4753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.761486 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.761491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.761495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.761654 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.762153 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.762163 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.762166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.762585 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.762741 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.763195 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.763502 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 16:00:01.764735 24299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:01.765089 24299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } }
I20260924 16:00:01.766882 24299 leader_election.cc:288] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4167211be21948888b84c382f610a8b4 (127.16.233.190:36701), 85ffe92f9a0e479c940899f0a207cec3 (127.16.233.189:41823)
I20260924 16:00:01.766954 23565 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.233.188:46427: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a"
member_type: NON_VOTER
last_known_addr {
host: "127.16.233.188"
port: 46427
}
attrs {
promote: true
}
}
dest_uuid: "4167211be21948888b84c382f610a8b4"
cas_config_opid_index: 5
I20260924 16:00:01.767588 23576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4167211be21948888b84c382f610a8b4" is_pre_election: true
I20260924 16:00:01.768158 23565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }
Sep 24 16:00:01 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.129@KRBTEST.COM
I20260924 16:00:01.770144 24243 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c5044a027d640be8f94f8f5bc5ee50a (127.16.233.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.771394 24182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 LEARNER]: Refusing update from remote peer 4167211be21948888b84c382f610a8b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 16:00:01.772073 24321 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 16:00:01.775769 24266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:01.778919 24181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ffe92f9a0e479c940899f0a207cec3" is_pre_election: true
I20260924 16:00:01.779503 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.740111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.16.233.129:39563)
0924 16:00:01.755984 (+ 15873us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.755995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.756030 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:00:01.756117 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.756127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.756490 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.756673 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.758702 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.760610 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.762410 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.763412 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.764251 (+ 839us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.770540 (+ 6289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.770559 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.770578 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.770637 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.775040 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.776030 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.776040 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.776044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.776198 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.776709 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.776714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.776717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.777157 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.778607 (+ 1450us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.778781 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.779070 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15633,"thread_start_us":16149,"threads_started":1}
I20260924 16:00:01.779850 24240 leader_election.cc:302] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c5044a027d640be8f94f8f5bc5ee50a; no voters: 4167211be21948888b84c382f610a8b4, 85ffe92f9a0e479c940899f0a207cec3
I20260924 16:00:01.781301 23752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:01.786866 24266 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 FOLLOWER]: Refusing update from remote peer 4167211be21948888b84c382f610a8b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 16:00:01.787120 24299 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 16:00:01.788271 24182 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c5044a027d640be8f94f8f5bc5ee50a (127.16.233.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.789059 24243 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.233.188:46427
I20260924 16:00:01.795979 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.796664 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:00:01.797079 24320 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:01.799861 24321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.800253 24321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:01.801404 24320 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: attempting to promote NON_VOTER 85ffe92f9a0e479c940899f0a207cec3 to VOTER
I20260924 16:00:01.803781 24320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }
I20260924 16:00:01.806098 24265 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 FOLLOWER]: Refusing update from remote peer 4167211be21948888b84c382f610a8b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 16:00:01.806216 24182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 LEARNER]: Refusing update from remote peer 4167211be21948888b84c382f610a8b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 16:00:01.806847 24243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 16:00:01.807349 24320 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:01.809624 24243 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85ffe92f9a0e479c940899f0a207cec3 (127.16.233.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.812557 24243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.812784 24243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:01.813279 24320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.813747 24320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:01.813866 24182 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85ffe92f9a0e479c940899f0a207cec3 (127.16.233.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.814841 24133 master.cc:537] Master@127.16.233.188:46427 shutting down...
W20260924 16:00:01.814898 24242 tablet.cc:2369] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:00:01.821015 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.821425 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:00:01.824402 24266 raft_consensus.cc:858] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }
W20260924 16:00:01.824620 24266 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }
I20260924 16:00:01.839570 24266 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 85ffe92f9a0e479c940899f0a207cec3 (127.16.233.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } }
I20260924 16:00:01.847056 24265 raft_consensus.cc:1467] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260924 16:00:01.847313 24328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } }
I20260924 16:00:01.848031 24328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 16:00:01.847831 24265 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 NON_PARTICIPANT]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.5->[1.6-1.7]
I20260924 16:00:01.848866 24265 raft_consensus.cc:858] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }
W20260924 16:00:01.848992 24265 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }
I20260924 16:00:01.849375 24265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c5044a027d640be8f94f8f5bc5ee50a (127.16.233.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.853550 24299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4167211be21948888b84c382f610a8b4. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.853871 24299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: LEARNER
I20260924 16:00:01.853578 24331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.854278 24331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: LEARNER
I20260924 16:00:01.854238 24328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: NON_VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: true } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.854524 24328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: LEARNER
I20260924 16:00:01.856174 24320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:01.856459 24320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:01.858109 24133 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 LEARNER]: Raft consensus shutting down.
I20260924 16:00:01.858634 24133 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 LEARNER]: Raft consensus is shut down!
I20260924 16:00:01.858840 24133 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: stopping tablet replica
W20260924 16:00:01.860041 23552 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 -> Peer 1c5044a027d640be8f94f8f5bc5ee50a (127.16.233.188:46427): Couldn't send request to peer 1c5044a027d640be8f94f8f5bc5ee50a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 16:00:01.866364 24133 master.cc:552] Master@127.16.233.188:46427 shutdown complete.
I20260924 16:00:01.866469 24133 master_runner.cc:305] Clearing existing system tablet
I20260924 16:00:01.866822 24133 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:01.866933 24133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:01.869199 24133 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:00:01.870784 24334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:01.871105 24133 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:01.871387 24133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 16:00:01.871488 24133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal
uuid: "1c5044a027d640be8f94f8f5bc5ee50a"
format_stamp: "Formatted at 2026-09-24 16:00:01 on dist-test-slave-xn4h"
server_key: "56604e1ad3f5127f488ff6e6343afc6b"
server_key_iv: "c1a529d77e54e81607644788b2584ea9"
server_key_version: "encryptionkey@0"
I20260924 16:00:01.874737 24133 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:01.879483 24133 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:00:01.879664 24133 log.cc:1196] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 16:00:01.880182 24133 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Deleting consensus metadata
I20260924 16:00:01.880434 24133 master_runner.cc:315] Copying system tablet from 127.16.233.190:36701
I20260924 16:00:01.883049 24133 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.233.190:36701
I20260924 16:00:01.902949 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.883847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.16.233.190:36701)
0924 16:00:01.884149 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:01.884158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:01.890987 (+ 6829us) server_negotiation.cc:407] Connection header received
0924 16:00:01.891239 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:01.891243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:01.891318 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:01.891433 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:01.892320 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.893063 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.894348 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.894793 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.894931 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.896885 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:01.896890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:01.896893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:01.896926 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:01.899962 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.900746 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.900754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.900759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.900922 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:01.901359 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:01.901366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:01.901370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:01.901687 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:01.901853 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:01.902283 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 16:00:01.902556 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260924 16:00:01.903019 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:01.883780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54146)
0924 16:00:01.890836 (+ 7056us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:01.890859 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:01.891060 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:01.891468 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:01.891472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:01.891922 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:01.892147 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.892157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.893217 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.893222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.894171 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:01.894184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:01.894960 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:01.894966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:01.895201 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:01.895886 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:01.895904 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:01.896675 (+ 771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:01.900108 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.900113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.900135 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.900641 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.901072 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:01.901077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:01.901081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:01.901243 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:01.901825 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:01.901830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:01.902029 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 16:00:01.902183 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 16:00:01.902575 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6903,"thread_start_us":6769,"threads_started":1}
I20260924 16:00:01.903970 23587 tablet_copy_service.cc:140] P 4167211be21948888b84c382f610a8b4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1c5044a027d640be8f94f8f5bc5ee50a ({username='oryx', principal='oryx/127.16.233.188@KRBTEST.COM'} at 127.0.0.1:54146)
I20260924 16:00:01.904168 23587 tablet_copy_service.cc:161] P 4167211be21948888b84c382f610a8b4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1c5044a027d640be8f94f8f5bc5ee50a at {username='oryx', principal='oryx/127.16.233.188@KRBTEST.COM'} at 127.0.0.1:54146: session id = 1c5044a027d640be8f94f8f5bc5ee50a-00000000000000000000000000000000
I20260924 16:00:01.907156 23587 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:00:01.908910 24133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:00:01.913691 24133 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: tablet copy: Starting download of 0 data blocks...
I20260924 16:00:01.913986 24133 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: tablet copy: Starting download of 1 WAL segments...
I20260924 16:00:01.916761 24133 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:00:01.920176 23587 tablet_copy_service.cc:342] P 4167211be21948888b84c382f610a8b4: Request end of tablet copy session 1c5044a027d640be8f94f8f5bc5ee50a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.233.188@KRBTEST.COM'} at 127.0.0.1:54146
I20260924 16:00:01.920337 23587 tablet_copy_service.cc:433] P 4167211be21948888b84c382f610a8b4: ending tablet copy session 1c5044a027d640be8f94f8f5bc5ee50a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1c5044a027d640be8f94f8f5bc5ee50a
I20260924 16:00:01.931947 24133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:01.935232 24341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:01.936889 24342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:01.946041 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 23628
W20260924 16:00:01.950769 24344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:01.951500 24133 server_base.cc:1034] running on GCE node
I20260924 16:00:01.951838 24133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:01.952540 24133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:01.953737 24133 hybrid_clock.cc:648] HybridClock initialized: now 1790265601953668 us; error 65 us; skew 500 ppm
I20260924 16:00:01.953965 24133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:01 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:01.955940 24133 init.cc:377] Logged in from keytab as oryx/127.16.233.188@KRBTEST.COM (short username oryx)
I20260924 16:00:01.964154 24133 webserver.cc:460] Webserver started at http://127.16.233.188:46229/ using document root <none> and password file <none>
I20260924 16:00:01.964519 24133 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:01.964648 24133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:01.967222 24133 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:00:01.968971 24351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:01.969429 24133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 16:00:01.969583 24133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal
uuid: "1c5044a027d640be8f94f8f5bc5ee50a"
format_stamp: "Formatted at 2026-09-24 16:00:01 on dist-test-slave-xn4h"
server_key: "56604e1ad3f5127f488ff6e6343afc6b"
server_key_iv: "c1a529d77e54e81607644788b2584ea9"
server_key_version: "encryptionkey@0"
I20260924 16:00:01.969785 24133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:01.970809 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:39563
--local_ip_for_outbound_sockets=127.16.233.129
--tserver_master_addrs=127.16.233.190:36701
--webserver_port=40347
--webserver_interface=127.16.233.129
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab
--principal=oryx/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.233.190:36701,127.16.233.189:41823
--tserver_master_addrs=127.16.233.190:36701,127.16.233.189:41823,127.16.233.188:46427 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:00:01.985517 24133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:01.989332 24133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:02.029505 24133 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.188:46427
I20260924 16:00:02.029767 24403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.188:46427 every 8 connection(s)
I20260924 16:00:02.030654 24133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
I20260924 16:00:02.038488 24404 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:00:02.043560 24404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Bootstrap starting.
I20260924 16:00:02.056802 24404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:00:02.057425 24404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: Bootstrap complete.
I20260924 16:00:02.058293 24404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }
I20260924 16:00:02.058602 24404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c5044a027d640be8f94f8f5bc5ee50a, State: Initialized, Role: LEARNER
I20260924 16:00:02.058856 24404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } }
I20260924 16:00:02.060588 24407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:02.060961 24407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: LEARNER
I20260924 16:00:02.060999 24404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:00:02.084447 24419 catalog_manager.cc:1259] Loaded cluster ID: 13b7d706a32a44ed856c10eab84e6939
I20260924 16:00:02.084585 24419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: loading cluster ID for follower catalog manager: success
I20260924 16:00:02.087330 24419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: acquiring CA information for follower catalog manager: success
I20260924 16:00:02.089187 24419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 16:00:02.401068 24356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:02.401377 24356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:02.401540 24356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:02.409284 24356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:02.409407 24356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:02.409598 24356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:00:02.422066 24356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab
--principal=oryx/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:39563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=40347
--tserver_master_addrs=127.16.233.190:36701,127.16.233.189:41823,127.16.233.188:46427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:02.423679 24356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:02.425194 24356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:02.440768 24425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:02.440721 24426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:02.440735 24428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:02.441717 24356 server_base.cc:1034] running on GCE node
I20260924 16:00:02.442260 24356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:02.443307 24356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:02.444535 24356 hybrid_clock.cc:648] HybridClock initialized: now 1790265602444489 us; error 54 us; skew 500 ppm
I20260924 16:00:02.444849 24356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:02 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265602, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:02.448151 24356 init.cc:377] Logged in from keytab as oryx/127.16.233.129@KRBTEST.COM (short username oryx)
I20260924 16:00:02.449625 24356 webserver.cc:460] Webserver started at http://127.16.233.129:40347/ using document root <none> and password file <none>
I20260924 16:00:02.450345 24356 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:02.450519 24356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:02.455714 24356 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 16:00:02.458498 24435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:02.459815 24356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:00:02.459997 24356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "9f72561ef010464faf6a7ae00ab1ec7a"
format_stamp: "Formatted at 2026-09-24 15:59:59 on dist-test-slave-xn4h"
server_key: "521f4bcb26eeb2a8ac22e9adf4345122"
server_key_iv: "4f68437b4f3f06fb997ae82cf015d529"
server_key_version: "encryptionkey@0"
I20260924 16:00:02.461050 24356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:02.464042 24420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:02.464349 24420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:02.473027 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:02.464344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:46427 (local address 127.0.0.1:40038)
0924 16:00:02.464884 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:02.464905 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:02.465046 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:02.465959 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:02.465963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:02.465989 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:02.466373 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:02.466386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.468766 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.468770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:02.470046 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:02.470085 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.472450 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.472455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:02.472596 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:02.472621 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:00:02.472686 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 16:00:02.472792 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":211,"threads_started":1}
I20260924 16:00:02.473477 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:02.464640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.16.233.188:46427)
0924 16:00:02.465557 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:02.465562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:02.465582 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:00:02.465663 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:02.465669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:02.465760 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:02.465913 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:02.466561 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.468519 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.470296 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.472237 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.472579 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:02.472752 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:02.472804 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 16:00:02.472973 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":652,"threads_started":1}
I20260924 16:00:02.474227 24379 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 16:00:02.474650 24407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4167211be21948888b84c382f610a8b4. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: NON_VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: true } } }
I20260924 16:00:02.474947 24407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: LEARNER
I20260924 16:00:02.475333 24420 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: attempting to promote NON_VOTER 1c5044a027d640be8f94f8f5bc5ee50a to VOTER
I20260924 16:00:02.475932 24420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } }
I20260924 16:00:02.477686 24379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 LEARNER]: Refusing update from remote peer 4167211be21948888b84c382f610a8b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 16:00:02.477820 24182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Refusing update from remote peer 4167211be21948888b84c382f610a8b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 16:00:02.478224 24437 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 16:00:02.478518 24436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 16:00:02.481388 24437 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c5044a027d640be8f94f8f5bc5ee50a (127.16.233.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:02.481770 24182 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c5044a027d640be8f94f8f5bc5ee50a (127.16.233.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:02.481935 24379 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c5044a027d640be8f94f8f5bc5ee50a (127.16.233.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:02.485098 24437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:02.485353 24437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:02.485379 24436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:02.485651 24436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:02.485683 24437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:02.485867 24356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:02.486086 24437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:02.487146 24407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:02.487506 24407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:00:02.487754 24446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4167211be21948888b84c382f610a8b4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:02.488059 24446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:00:02.493024 24356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:02.493340 24356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:02.494693 24356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:02.496243 24356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:02.496366 24356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:02.496495 24356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:02.496587 24356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:02.541075 24356 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:39563
I20260924 16:00:02.541055 24556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:39563 every 8 connection(s)
I20260924 16:00:02.543128 24356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:00:02.549469 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 24356
I20260924 16:00:02.549887 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 23763
Sep 24 16:00:02 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265602, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.189@KRBTEST.COM
Sep 24 16:00:02 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265602, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:02.589581 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:02.547107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:43269 (local address 127.16.233.189:41823)
0924 16:00:02.562880 (+ 15773us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:02.562886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:02.562908 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:00:02.565277 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:02.565284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:02.565377 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:02.565519 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:02.570342 (+ 4823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.571594 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.574705 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.575476 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.575562 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:02.581327 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:02.581336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:02.581341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:02.581387 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:02.585108 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:02.586082 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:02.586087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:02.586090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:02.586256 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:02.586833 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:02.586841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:02.586846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:02.587259 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:02.587390 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:02.588889 (+ 1499us) server_negotiation.cc:299] Negotiation successful
0924 16:00:02.589151 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15621,"thread_start_us":15835,"threads_started":1}
Sep 24 16:00:02 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265602, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.188@KRBTEST.COM
I20260924 16:00:02.590637 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:02.547002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:37127 (local address 127.16.233.190:36701)
0924 16:00:02.568014 (+ 21012us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:02.568022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:02.568075 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:00:02.568139 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:02.568152 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:02.568231 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:02.568346 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:02.570643 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.571779 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.575412 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.576051 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.576181 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:02.581825 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:02.581832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:02.581836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:02.581878 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:02.585940 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:02.587059 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:02.587064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:02.587068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:02.587265 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:02.587896 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:02.587902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:02.587906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:02.588261 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:02.588402 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:02.589933 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0924 16:00:02.590204 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20768,"thread_start_us":20375,"threads_started":1}
I20260924 16:00:02.591861 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:02.548204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.16.233.129:43269)
0924 16:00:02.562675 (+ 14471us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:02.562742 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:02.564977 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:02.566061 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:02.566076 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:02.568520 (+ 2444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:02.570036 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:02.570062 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.571857 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.571862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:02.574480 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:02.574502 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.575949 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.575957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:02.576246 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:02.577213 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:02.577254 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:02.581002 (+ 3748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:02.585297 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:02.585308 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:02.585324 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:02.585929 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:02.586443 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:02.586447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:02.586450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:02.586693 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:02.587513 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:02.587524 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:02.588421 (+ 897us) client_negotiation.cc:769] Sending connection context
0924 16:00:02.589789 (+ 1368us) client_negotiation.cc:240] Negotiation successful
0924 16:00:02.590373 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13887,"mutex_wait_us":226,"spinlock_wait_cycles":70400,"thread_start_us":9694,"threads_started":1}
I20260924 16:00:02.592906 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:02.548886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.129:37127)
0924 16:00:02.565646 (+ 16760us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:02.565666 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:02.565807 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:02.568574 (+ 2767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:02.568578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:02.569910 (+ 1332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:02.570277 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:02.570290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.571918 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.571929 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:02.575249 (+ 3320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:02.575262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.576693 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.576700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:02.576831 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:02.578391 (+ 1560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:02.578416 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:02.581589 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:02.586326 (+ 4737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:02.586334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:02.586339 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:02.586911 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:02.587596 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:02.587601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:02.587604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:02.587768 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:02.589389 (+ 1621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:02.589395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:02.589540 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 16:00:02.589738 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 16:00:02.590825 (+ 1087us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16588,"mutex_wait_us":1685,"thread_start_us":30799,"threads_started":1}
I20260924 16:00:02.593930 24560 heartbeater.cc:347] Connected to a master server at 127.16.233.189:41823
I20260924 16:00:02.594058 24559 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36701
I20260924 16:00:02.594287 24560 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:02.594288 24559 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:02.595409 24559 heartbeater.cc:511] Master 127.16.233.190:36701 requested a full tablet report, sending...
I20260924 16:00:02.595417 24560 heartbeater.cc:511] Master 127.16.233.189:41823 requested a full tablet report, sending...
Sep 24 16:00:02 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.130@KRBTEST.COM
I20260924 16:00:02.597440 23565 ts_manager.cc:194] Re-registered known tserver with Master: 9f72561ef010464faf6a7ae00ab1ec7a (127.16.233.129:39563)
I20260924 16:00:02.598112 24172 ts_manager.cc:194] Registered new tserver with Master: 9f72561ef010464faf6a7ae00ab1ec7a (127.16.233.129:39563)
I20260924 16:00:02.598788 23565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:37127
I20260924 16:00:02.599494 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:02.547629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:51645 (local address 127.16.233.188:46427)
0924 16:00:02.547884 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:02.547890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:02.580279 (+ 32389us) server_negotiation.cc:407] Connection header received
0924 16:00:02.580435 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:02.580444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:02.580542 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:02.580713 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:02.582632 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.583394 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.585335 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.586023 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.586119 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:02.592637 (+ 6518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:02.592646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:02.592654 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:02.592700 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:02.596274 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:02.597146 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:02.597152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:02.597156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:02.597332 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:02.597804 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:02.597808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:02.597811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:02.598173 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:02.598343 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:02.598876 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 16:00:02.599142 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 16:00:02.600394 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:02.552243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:46427 (local address 127.16.233.129:51645)
0924 16:00:02.580137 (+ 27894us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:02.580158 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:02.580296 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:02.581038 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:02.581045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:02.581762 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:02.582074 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:02.582085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.583598 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.583603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:02.585176 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:02.585192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.587133 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.587140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:02.587325 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:02.587966 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:02.587986 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:02.592405 (+ 4419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:02.596448 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:02.596453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:02.596457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:02.596967 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:02.597490 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:02.597496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:02.597500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:02.597677 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:02.598425 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:02.598430 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:02.598595 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 16:00:02.599796 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0924 16:00:02.600073 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27723,"mutex_wait_us":2,"thread_start_us":7599,"threads_started":1}
I20260924 16:00:02.601449 24558 heartbeater.cc:347] Connected to a master server at 127.16.233.188:46427
I20260924 16:00:02.601611 24558 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:02.602052 24558 heartbeater.cc:511] Master 127.16.233.188:46427 requested a full tablet report, sending...
I20260924 16:00:02.604029 24369 ts_manager.cc:194] Registered new tserver with Master: 9f72561ef010464faf6a7ae00ab1ec7a (127.16.233.129:39563)
I20260924 16:00:02.605520 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:02.551460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.16.233.130:46313)
0924 16:00:02.578714 (+ 27254us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:02.578728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:02.581795 (+ 3067us) server_negotiation.cc:407] Connection header received
0924 16:00:02.581877 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:02.581885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:02.582173 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:02.582356 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:02.585591 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.586942 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.588792 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:02.589392 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:02.589531 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:02.597565 (+ 8034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:02.597585 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:02.597601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:02.597644 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:02.601773 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:02.602742 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:02.602775 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:02.602778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:02.602976 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:02.603398 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:02.603404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:02.603410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:02.603844 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:02.604159 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:02.605038 (+ 879us) server_negotiation.cc:299] Negotiation successful
0924 16:00:02.605238 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26919,"thread_start_us":27281,"threads_started":1}
I20260924 16:00:02.606791 23886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:02.695204 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 23763
I20260924 16:00:02.712344 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:46313
--local_ip_for_outbound_sockets=127.16.233.130
--tserver_master_addrs=127.16.233.190:36701
--webserver_port=38903
--webserver_interface=127.16.233.130
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab
--principal=oryx/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.233.190:36701,127.16.233.189:41823
--tserver_master_addrs=127.16.233.190:36701,127.16.233.189:41823,127.16.233.188:46427 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:03.049108 24570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:03.049383 24570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:03.049475 24570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:03.056711 24570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:03.056852 24570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:03.056969 24570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:00:03.067267 24570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab
--principal=oryx/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:46313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=38903
--tserver_master_addrs=127.16.233.190:36701,127.16.233.189:41823,127.16.233.188:46427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:03.068627 24570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:03.070082 24570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:03.083468 24576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:03.083611 24575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:03.083611 24578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:03.084638 24570 server_base.cc:1034] running on GCE node
I20260924 16:00:03.085224 24570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:03.086127 24570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:03.087369 24570 hybrid_clock.cc:648] HybridClock initialized: now 1790265603087324 us; error 57 us; skew 500 ppm
I20260924 16:00:03.087693 24570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:03 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265603, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:03.090930 24570 init.cc:377] Logged in from keytab as oryx/127.16.233.130@KRBTEST.COM (short username oryx)
I20260924 16:00:03.092496 24570 webserver.cc:460] Webserver started at http://127.16.233.130:38903/ using document root <none> and password file <none>
I20260924 16:00:03.093231 24570 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:03.093405 24570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:03.098966 24570 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:00:03.101852 24585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:03.103298 24570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:03.103500 24570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "7ef64d9eec314c388971bcb822442cdb"
format_stamp: "Formatted at 2026-09-24 15:59:59 on dist-test-slave-xn4h"
server_key: "c6239b428a31179ff7735f948f6c81f0"
server_key_iv: "379a90b195d1498368bcab8b1104c9d9"
server_key_version: "encryptionkey@0"
I20260924 16:00:03.104529 24570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:03.126971 24570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:03.131512 24570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:03.131865 24570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:03.133061 24570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:03.134543 24570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:03.134668 24570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:03.134804 24570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:03.134896 24570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:03.168664 24570 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:46313
I20260924 16:00:03.168723 24698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:46313 every 8 connection(s)
I20260924 16:00:03.170252 24570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:00:03.172921 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 24570
I20260924 16:00:03.173219 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 23896
Sep 24 16:00:03 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265603, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.189@KRBTEST.COM
Sep 24 16:00:03 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265603, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.188@KRBTEST.COM
Sep 24 16:00:03 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.131@KRBTEST.COM
I20260924 16:00:03.226177 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.173542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:59385 (local address 127.16.233.188:46427)
0924 16:00:03.185820 (+ 12278us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.185831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.197793 (+ 11962us) server_negotiation.cc:407] Connection header received
0924 16:00:03.205947 (+ 8154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.205954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.206039 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.206117 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.207275 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.208425 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.213072 (+ 4647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.213768 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.213902 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.218109 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.218114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.218117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.218152 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.221926 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.222991 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.222998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.223004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.223303 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.223896 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.223901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.223905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.224480 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.224706 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.225576 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.225903 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12093,"thread_start_us":11978,"threads_started":1}
Sep 24 16:00:03 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265603, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:03.227579 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.174702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:52845 (local address 127.16.233.189:41823)
0924 16:00:03.196462 (+ 21760us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.196473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.202194 (+ 5721us) server_negotiation.cc:407] Connection header received
0924 16:00:03.202259 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.202265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.202383 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.202476 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.206628 (+ 4152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.208072 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.211410 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.212079 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.212208 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.217291 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.217298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.217303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.217359 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.220967 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.222151 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.222156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.222176 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.222396 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.223488 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.223496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.223501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.224031 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.224172 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.226511 (+ 2339us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.226816 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21555,"thread_start_us":21794,"threads_started":1}
I20260924 16:00:03.228173 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.173922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:46427 (local address 127.16.233.130:59385)
0924 16:00:03.197506 (+ 23584us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:03.197585 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:03.200417 (+ 2832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:03.206374 (+ 5957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:03.206379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:03.206834 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:03.207101 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.207111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.208701 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.208706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.212908 (+ 4202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.212921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.214234 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.214245 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.214528 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.215111 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:03.215134 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:03.217850 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:03.222089 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.222095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.222098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.222777 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.223460 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.223466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.223470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.223696 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.224819 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:03.224835 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:03.225016 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 16:00:03.226448 (+ 1432us) client_negotiation.cc:240] Negotiation successful
0924 16:00:03.226773 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23062,"mutex_wait_us":2356,"thread_start_us":31514,"threads_started":1}
I20260924 16:00:03.228271 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.174210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.16.233.130:52845)
0924 16:00:03.201867 (+ 27657us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:03.201884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:03.201998 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:03.204092 (+ 2094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:03.204108 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:03.205281 (+ 1173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:03.206355 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.206389 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.208311 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.208319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.211191 (+ 2872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.211222 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.212528 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.212536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.212901 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.213763 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:03.213799 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:03.217031 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:03.221159 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.221169 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.221191 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.222008 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.222565 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.222569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.222573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.222855 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.224264 (+ 1409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:03.224278 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:03.224885 (+ 607us) client_negotiation.cc:769] Sending connection context
0924 16:00:03.225327 (+ 442us) client_negotiation.cc:240] Negotiation successful
0924 16:00:03.225945 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27517,"mutex_wait_us":17,"thread_start_us":28644,"threads_started":1}
I20260924 16:00:03.230094 24701 heartbeater.cc:347] Connected to a master server at 127.16.233.189:41823
I20260924 16:00:03.230232 24699 heartbeater.cc:347] Connected to a master server at 127.16.233.188:46427
I20260924 16:00:03.230608 24699 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:03.230615 24701 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:03.231853 24699 heartbeater.cc:511] Master 127.16.233.188:46427 requested a full tablet report, sending...
I20260924 16:00:03.232200 24701 heartbeater.cc:511] Master 127.16.233.189:41823 requested a full tablet report, sending...
I20260924 16:00:03.233996 24172 ts_manager.cc:194] Registered new tserver with Master: 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313)
I20260924 16:00:03.234845 24369 ts_manager.cc:194] Registered new tserver with Master: 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313)
I20260924 16:00:03.236011 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.174250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.16.233.131:45151)
0924 16:00:03.209512 (+ 35262us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.209521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.209549 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:00:03.209610 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.209617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.209871 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.210009 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.212751 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.214410 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.216455 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.217386 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.217529 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.225585 (+ 8056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.225604 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.225621 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.225670 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.231411 (+ 5741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.232696 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.232723 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.232726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.233016 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.233471 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.233476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.233480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.233955 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.235274 (+ 1319us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.235440 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.235684 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35094,"thread_start_us":34935,"threads_started":1}
I20260924 16:00:03.236565 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.174516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.130:34695)
0924 16:00:03.212421 (+ 37905us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:03.212443 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:03.212566 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:03.213100 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:03.213104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:03.214141 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:03.214576 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.214591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.217592 (+ 3001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.217597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.220149 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.220166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.222052 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.222061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.222453 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.223225 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:03.223242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:03.227336 (+ 4094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:03.233372 (+ 6036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.233377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.233380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.233867 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.234359 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.234364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.234367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.234541 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.235564 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:03.235569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:03.235737 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 16:00:03.235872 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 16:00:03.236161 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37763,"thread_start_us":37590,"threads_started":1}
I20260924 16:00:03.236682 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.174349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:34695 (local address 127.16.233.190:36701)
0924 16:00:03.202912 (+ 28563us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.202919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.212555 (+ 9636us) server_negotiation.cc:407] Connection header received
0924 16:00:03.212697 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.212701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.212791 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.212919 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.215031 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.217431 (+ 2400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.220340 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.221445 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.221565 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.227580 (+ 6015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.227587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.227592 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.227637 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.233210 (+ 5573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.234025 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.234030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.234034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.234187 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.235046 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.235051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.235054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.235420 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.235962 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.236102 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.236313 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28437,"thread_start_us":28354,"threads_started":1}
I20260924 16:00:03.237434 24700 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36701
I20260924 16:00:03.237455 24021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:03.237576 24700 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:03.238134 24700 heartbeater.cc:511] Master 127.16.233.190:36701 requested a full tablet report, sending...
I20260924 16:00:03.239346 23565 ts_manager.cc:194] Re-registered known tserver with Master: 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313)
I20260924 16:00:03.240868 23565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:34695
I20260924 16:00:03.321946 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 23896
I20260924 16:00:03.338934 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:45151
--local_ip_for_outbound_sockets=127.16.233.131
--tserver_master_addrs=127.16.233.190:36701
--webserver_port=43561
--webserver_interface=127.16.233.131
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab
--principal=oryx/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.233.190:36701,127.16.233.189:41823
--tserver_master_addrs=127.16.233.190:36701,127.16.233.189:41823,127.16.233.188:46427 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:00:03.602200 24559 heartbeater.cc:503] Master 127.16.233.190:36701 was elected leader, sending a full tablet report...
W20260924 16:00:03.677891 24712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:03.678237 24712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:03.678431 24712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:03.685916 24712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:03.686081 24712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:03.686265 24712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:00:03.696797 24712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab
--principal=oryx/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:45151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=43561
--tserver_master_addrs=127.16.233.190:36701,127.16.233.189:41823,127.16.233.188:46427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:03.698182 24712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:03.699576 24712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:03.713833 24717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:03.713958 24720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:03.713838 24718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:03.714603 24712 server_base.cc:1034] running on GCE node
I20260924 16:00:03.715221 24712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:03.716176 24712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:03.717413 24712 hybrid_clock.cc:648] HybridClock initialized: now 1790265603717369 us; error 34 us; skew 500 ppm
I20260924 16:00:03.717760 24712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:03 dist-test-slave-xn4h krb5kdc[23521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265603, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:03.741699 24712 init.cc:377] Logged in from keytab as oryx/127.16.233.131@KRBTEST.COM (short username oryx)
I20260924 16:00:03.743525 24712 webserver.cc:460] Webserver started at http://127.16.233.131:43561/ using document root <none> and password file <none>
I20260924 16:00:03.744417 24712 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:03.744599 24712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:03.750485 24712 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 16:00:03.753166 24728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:03.754657 24712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:03.754823 24712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "ec6e7c31a45049689e8e72ade7a08763"
format_stamp: "Formatted at 2026-09-24 16:00:00 on dist-test-slave-xn4h"
server_key: "e74f8e823a94dc45ee5daf6549f4cf2a"
server_key_iv: "3475d4e0d2d378a08db95d2c2ffeb582"
server_key_version: "encryptionkey@0"
I20260924 16:00:03.755836 24712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:03.786615 24712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:03.790866 24712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:03.791159 24712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:03.792152 24712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:03.793537 24712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:03.793648 24712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:03.793779 24712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:03.793881 24712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:03.827325 24712 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:45151
I20260924 16:00:03.827389 24841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:45151 every 8 connection(s)
I20260924 16:00:03.829013 24712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:00:03.831164 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 24712
Sep 24 16:00:03 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265603, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
Sep 24 16:00:03 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265603, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.189@KRBTEST.COM
I20260924 16:00:03.873466 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.832944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.131:42425)
0924 16:00:03.834024 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:03.834089 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:03.840279 (+ 6190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:03.850978 (+ 10699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:03.850982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:03.851510 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:03.852172 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.852182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.853539 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.853546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.856208 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.856220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.857560 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.857568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.858035 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.859180 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:03.859222 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:03.862417 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:03.867469 (+ 5052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.867482 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.867506 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.868240 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.868772 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.868779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.868785 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.869061 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.869854 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:03.869867 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:03.870492 (+ 625us) client_negotiation.cc:769] Sending connection context
0924 16:00:03.870878 (+ 386us) client_negotiation.cc:240] Negotiation successful
0924 16:00:03.871612 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":616,"spinlock_wait_cycles":20352,"thread_start_us":1214,"threads_started":1}
I20260924 16:00:03.875124 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.832228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:42425 (local address 127.16.233.190:36701)
0924 16:00:03.850679 (+ 18451us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.850684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.850700 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:00:03.850752 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.850755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.850811 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.850947 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.852349 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.853374 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.856399 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.857082 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.857205 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.862697 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.862704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.862708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.862751 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.867244 (+ 4493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.868425 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.868434 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.868440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.868625 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.869212 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.869219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.869225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.869604 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.869746 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.874267 (+ 4521us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.874631 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18343,"thread_start_us":18235,"threads_started":1}
I20260924 16:00:03.876643 24843 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36701
I20260924 16:00:03.877063 24843 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:03.877985 24843 heartbeater.cc:511] Master 127.16.233.190:36701 requested a full tablet report, sending...
I20260924 16:00:03.880596 23565 ts_manager.cc:194] Re-registered known tserver with Master: ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151)
I20260924 16:00:03.882031 23565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:42425
I20260924 16:00:03.893343 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.833904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.16.233.131:49965)
0924 16:00:03.846929 (+ 13025us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:03.846955 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:03.847131 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:03.848579 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:03.848598 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:03.849890 (+ 1292us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:03.851954 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.851983 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.853498 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.853505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.856205 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.856220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.857560 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.857568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.858032 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.859262 (+ 1230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:03.859297 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:03.872203 (+ 12906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:03.888276 (+ 16073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.888285 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.888291 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.889673 (+ 1382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.890343 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.890350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.890356 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.890627 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.891568 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:03.891575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:03.891857 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 16:00:03.892054 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 16:00:03.892431 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12838,"mutex_wait_us":312,"thread_start_us":12627,"threads_started":1}
I20260924 16:00:03.895239 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.833820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:49965 (local address 127.16.233.189:41823)
0924 16:00:03.847632 (+ 13812us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.847639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.847659 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:00:03.847739 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.847747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.847851 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.847978 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.852159 (+ 4181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.853278 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.856393 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.857080 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.857205 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.872498 (+ 15293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.872506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.872511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.872583 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.888076 (+ 15493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.889894 (+ 1818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.889904 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.889909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.890175 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.890801 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.890807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.890813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.891407 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.893973 (+ 2566us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.894186 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.894536 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13640,"thread_start_us":13535,"threads_started":1}
I20260924 16:00:03.899205 24844 heartbeater.cc:347] Connected to a master server at 127.16.233.189:41823
I20260924 16:00:03.900100 24844 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:03.901232 24844 heartbeater.cc:511] Master 127.16.233.189:41823 requested a full tablet report, sending...
I20260924 16:00:03.903296 24172 ts_manager.cc:194] Registered new tserver with Master: ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151)
Sep 24 16:00:03 dist-test-slave-xn4h krb5kdc[23521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265603, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.188@KRBTEST.COM
I20260924 16:00:03.912581 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.863050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.16.233.188:46427)
0924 16:00:03.878605 (+ 15555us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.878614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.878640 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:00:03.878728 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.878736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.878837 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.882313 (+ 3476us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.882671 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.884266 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.886905 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.887946 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.888102 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.897266 (+ 9164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.897275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.897283 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.897330 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.904225 (+ 6895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.906742 (+ 2517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.906748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.906751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.908503 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.909184 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.909192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.909197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.910240 (+ 1043us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.910591 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.911931 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.912175 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":15381,"spinlock_wait_cycles":11008,"thread_start_us":15934,"threads_started":1}
I20260924 16:00:03.913574 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.841667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:46427 (local address 127.16.233.131:59889)
0924 16:00:03.858006 (+ 16339us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:03.858029 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:03.858161 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:03.858693 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:03.858701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:03.859464 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:03.859877 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.859890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.861467 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.861474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.874405 (+ 12931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.874424 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.895674 (+ 21250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.895685 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.896087 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.897124 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:03.897155 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:03.904537 (+ 7382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:03.910135 (+ 5598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.910142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.910148 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.910731 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.911215 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:03.911222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:03.911228 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:03.911409 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:03.912083 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:03.912088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:03.912261 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 16:00:03.912381 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 16:00:03.912690 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16203,"thread_start_us":16077,"threads_started":1}
I20260924 16:00:03.915519 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.841612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:59889 (local address 127.16.233.188:46427)
0924 16:00:03.857484 (+ 15872us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.857493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.858272 (+ 779us) server_negotiation.cc:407] Connection header received
0924 16:00:03.858403 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.858412 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.858520 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.858629 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.860072 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.861319 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.887402 (+ 26083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.889773 (+ 2371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.896649 (+ 6876us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.904814 (+ 8165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.904873 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.904879 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.904957 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.909961 (+ 5004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.910910 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.910914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.910917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.911069 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.911553 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.911559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.911564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.911938 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.913955 (+ 2017us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.914076 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.914980 (+ 904us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1260,"server-negotiator.queue_time_us":15693,"thread_start_us":16385,"threads_started":1}
I20260924 16:00:03.916934 24842 heartbeater.cc:347] Connected to a master server at 127.16.233.188:46427
I20260924 16:00:03.917101 24842 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:03.917459 24842 heartbeater.cc:511] Master 127.16.233.188:46427 requested a full tablet report, sending...
I20260924 16:00:03.919090 24369 ts_manager.cc:194] Registered new tserver with Master: ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151)
I20260924 16:00:03.924631 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.862784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.16.233.190:36701)
0924 16:00:03.876745 (+ 13961us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.876751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.903400 (+ 26649us) server_negotiation.cc:407] Connection header received
0924 16:00:03.906383 (+ 2983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.906390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.906546 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.906629 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.909706 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.910897 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.912849 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.913395 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.913497 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.917681 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.917689 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.917696 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.917743 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.921521 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.922160 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.922167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.922171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.922351 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.922858 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.922862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.922864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.923149 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.923567 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.924002 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.924263 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13832,"thread_start_us":42340,"threads_started":1}
I20260924 16:00:03.925623 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.869602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.16.233.189:41823)
0924 16:00:03.899705 (+ 30103us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.899712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.908423 (+ 8711us) server_negotiation.cc:407] Connection header received
0924 16:00:03.908949 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.908956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.909033 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.909111 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:03.911480 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.912664 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.914573 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.915314 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.915466 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.918949 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:03.918956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:03.918960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:03.919007 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:03.922371 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.923285 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.923289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.923292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.923445 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:03.923978 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:03.923983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:03.923986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:03.924321 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:03.924553 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.925023 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.925264 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29928,"thread_start_us":30122,"threads_started":1}
I20260924 16:00:03.929095 23565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:00:03.936519 23565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:00:03.966672 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.955479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:45151 (local address 127.0.0.1:47740)
0924 16:00:03.958703 (+ 3224us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:03.958718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:03.958877 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:03.959889 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:03.959893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:03.959919 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:03.960398 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.960415 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.962350 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.962358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.963912 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.963924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.965781 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.965789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.966029 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.966054 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:00:03.966162 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:00:03.966292 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3124,"thread_start_us":3006,"threads_started":1}
I20260924 16:00:03.967089 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.956129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.16.233.131:45151)
0924 16:00:03.956882 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.956891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.958824 (+ 1933us) server_negotiation.cc:407] Connection header received
0924 16:00:03.959404 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.959412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.959713 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.959905 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:03.960591 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.962198 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.964050 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.965103 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.965274 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.965482 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.966210 (+ 728us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.966417 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":1007,"threads_started":1}
I20260924 16:00:03.975534 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.956528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:39563 (local address 127.0.0.1:50732)
0924 16:00:03.966926 (+ 10398us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:03.966944 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:03.967145 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:03.968050 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:03.968053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:03.968073 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:03.968187 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.968193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.969601 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.969609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.972497 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.972512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.974704 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.974710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.974904 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.974936 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:00:03.975024 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 16:00:03.975167 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10305,"thread_start_us":15421,"threads_started":1}
I20260924 16:00:03.975585 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.956797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.16.233.129:39563)
0924 16:00:03.957322 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.957332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.967032 (+ 9700us) server_negotiation.cc:407] Connection header received
0924 16:00:03.967349 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.967358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.967611 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.967748 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:03.968325 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.969471 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.972693 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.974274 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.974450 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.974616 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.975118 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.975290 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":678,"threads_started":1}
I20260924 16:00:03.974983 24776 tablet_service.cc:1467] Processing CreateTablet for tablet 15cab4df3cda4e8885f151832b810ef8 (DEFAULT_TABLE table=test-table [id=88fa665c51564fd0bc66354c35b94665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:03.976711 24776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15cab4df3cda4e8885f151832b810ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:03.979442 24491 tablet_service.cc:1467] Processing CreateTablet for tablet 15cab4df3cda4e8885f151832b810ef8 (DEFAULT_TABLE table=test-table [id=88fa665c51564fd0bc66354c35b94665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:03.980813 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15cab4df3cda4e8885f151832b810ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:03.981674 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.958839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:46313 (local address 127.0.0.1:54914)
0924 16:00:03.970735 (+ 11896us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:03.970752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:03.972203 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:03.973175 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:03.973180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:03.973203 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:03.973504 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.973516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.975289 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.975294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.977818 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:03.977834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.980624 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.980631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:03.980890 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.980928 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:00:03.981025 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:00:03.981157 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11766,"thread_start_us":11631,"threads_started":1}
I20260924 16:00:03.982534 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:03.959101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.16.233.130:46313)
0924 16:00:03.959562 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:03.959569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:03.972370 (+ 12801us) server_negotiation.cc:407] Connection header received
0924 16:00:03.972479 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:03.972487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:03.972735 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:03.972942 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:03.973652 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.975126 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.977990 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:03.979944 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:03.980195 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:03.980369 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:03.981965 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0924 16:00:03.982200 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":1129,"threads_started":1}
I20260924 16:00:03.986186 24634 tablet_service.cc:1467] Processing CreateTablet for tablet 15cab4df3cda4e8885f151832b810ef8 (DEFAULT_TABLE table=test-table [id=88fa665c51564fd0bc66354c35b94665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:03.987599 24634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15cab4df3cda4e8885f151832b810ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:03.994825 24882 tablet_bootstrap.cc:492] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: Bootstrap starting.
I20260924 16:00:03.997660 24882 tablet_bootstrap.cc:654] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:03.998817 24882 log.cc:824] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:03.998809 24881 tablet_bootstrap.cc:492] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: Bootstrap starting.
I20260924 16:00:04.002090 24882 tablet_bootstrap.cc:492] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: No bootstrap required, opened a new log
I20260924 16:00:04.002458 24882 ts_tablet_manager.cc:1397] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 16:00:04.002882 24881 tablet_bootstrap.cc:654] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:04.004106 24881 log.cc:824] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:04.006386 24885 tablet_bootstrap.cc:492] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: Bootstrap starting.
I20260924 16:00:04.007598 24881 tablet_bootstrap.cc:492] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: No bootstrap required, opened a new log
I20260924 16:00:04.007912 24881 ts_tablet_manager.cc:1397] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260924 16:00:04.010070 24885 tablet_bootstrap.cc:654] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:04.011327 24885 log.cc:824] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:04.013196 24882 raft_consensus.cc:348] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } }
I20260924 16:00:04.013635 24882 raft_consensus.cc:374] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:04.013732 24882 raft_consensus.cc:729] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f72561ef010464faf6a7ae00ab1ec7a, State: Initialized, Role: FOLLOWER
I20260924 16:00:04.014448 24882 consensus_queue.cc:260] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } }
I20260924 16:00:04.014703 24885 tablet_bootstrap.cc:492] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: No bootstrap required, opened a new log
I20260924 16:00:04.014982 24885 ts_tablet_manager.cc:1397] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 16:00:04.016443 24882 ts_tablet_manager.cc:1428] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 16:00:04.019619 24881 raft_consensus.cc:348] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } }
I20260924 16:00:04.020097 24881 raft_consensus.cc:374] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:04.020220 24881 raft_consensus.cc:729] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6e7c31a45049689e8e72ade7a08763, State: Initialized, Role: FOLLOWER
I20260924 16:00:04.021404 24881 consensus_queue.cc:260] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } }
I20260924 16:00:04.024346 24881 ts_tablet_manager.cc:1428] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20260924 16:00:04.024768 24843 heartbeater.cc:503] Master 127.16.233.190:36701 was elected leader, sending a full tablet report...
I20260924 16:00:04.029515 24885 raft_consensus.cc:348] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } }
I20260924 16:00:04.029954 24885 raft_consensus.cc:374] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:04.030079 24885 raft_consensus.cc:729] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef64d9eec314c388971bcb822442cdb, State: Initialized, Role: FOLLOWER
I20260924 16:00:04.030741 24885 consensus_queue.cc:260] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } }
I20260924 16:00:04.031977 24700 heartbeater.cc:503] Master 127.16.233.190:36701 was elected leader, sending a full tablet report...
I20260924 16:00:04.033382 24885 ts_tablet_manager.cc:1428] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
W20260924 16:00:04.079098 24561 tablet.cc:2369] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:00:04.092557 24845 tablet.cc:2369] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:00:04.173700 24702 tablet.cc:2369] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:00:04.245153 24887 raft_consensus.cc:482] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:04.245517 24887 raft_consensus.cc:504] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } }
I20260924 16:00:04.247359 24887 leader_election.cc:288] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151), 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313)
I20260924 16:00:04.254340 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:04.247613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:43931 (local address 127.16.233.131:45151)
0924 16:00:04.247886 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:04.247895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:04.248368 (+ 473us) server_negotiation.cc:407] Connection header received
0924 16:00:04.248621 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:04.248626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:04.248700 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:04.248768 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:04.249541 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.250532 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.252545 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.253364 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.253559 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:04.253635 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:04.253917 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 16:00:04.254047 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:00:04.254528 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:04.248303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:46313 (local address 127.16.233.129:47903)
0924 16:00:04.248811 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:04.248828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:04.248947 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:04.249487 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:04.249491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:04.249516 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:04.249839 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:04.249850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.251014 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.251018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:04.252490 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:04.252505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.253669 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.253676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:04.253842 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:04.253866 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:00:04.253967 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 16:00:04.254103 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":613,"threads_started":1}
I20260924 16:00:04.254544 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:04.248694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:47903 (local address 127.16.233.130:46313)
0924 16:00:04.248897 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:04.248903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:04.248924 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:00:04.249117 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:04.249121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:04.249179 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:04.249276 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:04.250010 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.250869 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.252645 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.253449 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.253626 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:04.253704 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:04.254005 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 16:00:04.254139 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:00:04.254782 24890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:04.247500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:45151 (local address 127.16.233.129:43931)
0924 16:00:04.248237 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:04.248261 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:04.248422 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:04.249010 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:04.249014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:04.249044 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:04.249391 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:04.249403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.250661 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.250667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:04.252351 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:04.252367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.253578 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.253584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:04.253757 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:04.253782 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:00:04.253889 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:00:04.254037 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":1,"thread_start_us":378,"threads_started":1}
I20260924 16:00:04.255548 24654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15cab4df3cda4e8885f151832b810ef8" candidate_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef64d9eec314c388971bcb822442cdb" is_pre_election: true
I20260924 16:00:04.255708 24796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15cab4df3cda4e8885f151832b810ef8" candidate_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6e7c31a45049689e8e72ade7a08763" is_pre_election: true
I20260924 16:00:04.256073 24654 raft_consensus.cc:2454] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f72561ef010464faf6a7ae00ab1ec7a in term 0.
I20260924 16:00:04.256217 24796 raft_consensus.cc:2454] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f72561ef010464faf6a7ae00ab1ec7a in term 0.
I20260924 16:00:04.256767 24443 leader_election.cc:302] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef64d9eec314c388971bcb822442cdb, 9f72561ef010464faf6a7ae00ab1ec7a; no voters:
I20260924 16:00:04.257121 24887 raft_consensus.cc:2781] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:00:04.257308 24887 raft_consensus.cc:482] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:00:04.257432 24887 raft_consensus.cc:3037] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:04.259647 24887 raft_consensus.cc:504] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } }
I20260924 16:00:04.260277 24887 leader_election.cc:288] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [CANDIDATE]: Term 1 election: Requested vote from peers ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151), 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313)
I20260924 16:00:04.260758 24796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15cab4df3cda4e8885f151832b810ef8" candidate_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6e7c31a45049689e8e72ade7a08763"
I20260924 16:00:04.260819 24654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15cab4df3cda4e8885f151832b810ef8" candidate_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef64d9eec314c388971bcb822442cdb"
I20260924 16:00:04.260967 24796 raft_consensus.cc:3037] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:04.261060 24654 raft_consensus.cc:3037] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:04.263355 24796 raft_consensus.cc:2454] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f72561ef010464faf6a7ae00ab1ec7a in term 1.
I20260924 16:00:04.263810 24443 leader_election.cc:302] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f72561ef010464faf6a7ae00ab1ec7a, ec6e7c31a45049689e8e72ade7a08763; no voters:
I20260924 16:00:04.264142 24887 raft_consensus.cc:2781] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:04.264444 24654 raft_consensus.cc:2454] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f72561ef010464faf6a7ae00ab1ec7a in term 1.
I20260924 16:00:04.264945 24887 raft_consensus.cc:686] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 LEADER]: Becoming Leader. State: Replica: 9f72561ef010464faf6a7ae00ab1ec7a, State: Running, Role: LEADER
I20260924 16:00:04.265530 24887 consensus_queue.cc:237] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } }
I20260924 16:00:04.271059 23566 catalog_manager.cc:5495] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f72561ef010464faf6a7ae00ab1ec7a (127.16.233.129). New cstate: current_term: 1 leader_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } health_report { overall_health: HEALTHY } } }
I20260924 16:00:04.332121 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:04.326572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.16.233.129:39563)
0924 16:00:04.326755 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:04.326759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:04.326954 (+ 195us) server_negotiation.cc:407] Connection header received
0924 16:00:04.327165 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:04.327169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:04.327255 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:04.327380 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:00:04.327690 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.328527 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.329893 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.330334 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.330424 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:04.330862 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:00:04.331481 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:00:04.331587 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:04.331700 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 16:00:04.331798 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:00:04.340526 24796 raft_consensus.cc:1261] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [term 1 FOLLOWER]: Refusing update from remote peer 9f72561ef010464faf6a7ae00ab1ec7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:04.340526 24654 raft_consensus.cc:1261] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [term 1 FOLLOWER]: Refusing update from remote peer 9f72561ef010464faf6a7ae00ab1ec7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:04.341605 24892 consensus_queue.cc:1035] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:04.341992 24887 consensus_queue.cc:1035] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:04.360044 24895 mvcc.cc:205] Tried to move back new op lower bound from 7332927915367788544 to 7332927915075653632. Current Snapshot: MvccSnapshot[applied={T|T < 7332927915367788544 or (T in {7332927915367788544})}]
I20260924 16:00:04.372846 23566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:04.373167 23566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:04.374961 24896 mvcc.cc:205] Tried to move back new op lower bound from 7332927915367788544 to 7332927915075653632. Current Snapshot: MvccSnapshot[applied={T|T < 7332927915367788544}]
I20260924 16:00:04.385062 23566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4167211be21948888b84c382f610a8b4: Sending DeleteTablet for 3 replicas of tablet 15cab4df3cda4e8885f151832b810ef8
I20260924 16:00:04.386461 24634 tablet_service.cc:1514] Processing DeleteTablet for tablet 15cab4df3cda4e8885f151832b810ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:04 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:54914
I20260924 16:00:04.386489 24775 tablet_service.cc:1514] Processing DeleteTablet for tablet 15cab4df3cda4e8885f151832b810ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:04 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:47740
I20260924 16:00:04.386513 24776 tablet_service.cc:1514] Processing DeleteTablet for tablet 15cab4df3cda4e8885f151832b810ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:04 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:47740
I20260924 16:00:04.387745 24906 tablet_replica.cc:330] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: stopping tablet replica
I20260924 16:00:04.388283 24906 raft_consensus.cc:2229] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:04.388579 24491 tablet_service.cc:1514] Processing DeleteTablet for tablet 15cab4df3cda4e8885f151832b810ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:04 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:50732
I20260924 16:00:04.388964 24906 raft_consensus.cc:2258] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:04.392138 24906 ts_tablet_manager.cc:1905] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:04.392956 24905 tablet_replica.cc:330] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: stopping tablet replica
I20260924 16:00:04.393337 24905 raft_consensus.cc:2229] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:04.393869 24905 raft_consensus.cc:2258] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:04.395539 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 23534
I20260924 16:00:04.396746 23597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:04.398062 24905 ts_tablet_manager.cc:1905] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:04.402243 24906 ts_tablet_manager.cc:1918] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:04.402606 24906 log.cc:1196] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/15cab4df3cda4e8885f151832b810ef8
I20260924 16:00:04.403617 24906 ts_tablet_manager.cc:1939] T 15cab4df3cda4e8885f151832b810ef8 P ec6e7c31a45049689e8e72ade7a08763: Deleting consensus metadata
I20260924 16:00:04.406013 23552 catalog_manager.cc:4841] TS ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151): tablet 15cab4df3cda4e8885f151832b810ef8 (table test-table [id=88fa665c51564fd0bc66354c35b94665]) successfully deleted
I20260924 16:00:04.406340 24905 ts_tablet_manager.cc:1918] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:04.406641 24905 log.cc:1196] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/15cab4df3cda4e8885f151832b810ef8
I20260924 16:00:04.407626 24905 ts_tablet_manager.cc:1939] T 15cab4df3cda4e8885f151832b810ef8 P 7ef64d9eec314c388971bcb822442cdb: Deleting consensus metadata
I20260924 16:00:04.414149 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:04.392009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.16.233.189:41823)
0924 16:00:04.392267 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:04.392277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:04.394822 (+ 2545us) server_negotiation.cc:407] Connection header received
0924 16:00:04.395056 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:04.395064 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:04.395157 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:04.395266 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:04.397684 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.398776 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.400707 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.401362 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.401457 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:04.405483 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:04.405491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:04.405496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:04.405539 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:04.409993 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:04.410876 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:04.410881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:04.410885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:04.411040 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:04.411548 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:04.411556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:04.411562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:04.412020 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:04.412307 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:04.412805 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 16:00:04.413081 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:00:04.419303 24907 tablet_replica.cc:330] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: stopping tablet replica
I20260924 16:00:04.419818 24907 raft_consensus.cc:2229] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:00:04.420586 24907 raft_consensus.cc:2258] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:04.424499 24907 ts_tablet_manager.cc:1905] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:04.430790 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:04.396130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.16.233.188:46427)
0924 16:00:04.396607 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:04.396615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:04.413394 (+ 16779us) server_negotiation.cc:407] Connection header received
0924 16:00:04.413604 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:04.413613 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:04.413720 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:04.413863 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:04.415919 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.417204 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.419314 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.419916 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.420046 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:04.423529 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:04.423535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:04.423539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:04.423584 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:04.427445 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:04.428307 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:04.428313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:04.428317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:04.428462 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:04.428932 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:04.428939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:04.428943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:04.429328 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:04.429643 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:04.430021 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 16:00:04.430349 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260924 16:00:04.433969 24907 ts_tablet_manager.cc:1918] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:04.434270 24907 log.cc:1196] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/15cab4df3cda4e8885f151832b810ef8
I20260924 16:00:04.435112 24907 ts_tablet_manager.cc:1939] T 15cab4df3cda4e8885f151832b810ef8 P 9f72561ef010464faf6a7ae00ab1ec7a: Deleting consensus metadata
W20260924 16:00:04.575394 23552 catalog_manager.cc:4808] TS ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151): delete failed for tablet 15cab4df3cda4e8885f151832b810ef8 because the tablet was not found. No further retry: Not found: Tablet not found: 15cab4df3cda4e8885f151832b810ef8
W20260924 16:00:04.575763 23552 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.233.131:45151, user_credentials={real_user=slave}} blocked reactor thread for 168421us
I20260924 16:00:04.576100 23551 catalog_manager.cc:4841] TS 9f72561ef010464faf6a7ae00ab1ec7a (127.16.233.129:39563): tablet 15cab4df3cda4e8885f151832b810ef8 (table test-table [id=88fa665c51564fd0bc66354c35b94665]) successfully deleted
I20260924 16:00:04.576391 23552 catalog_manager.cc:4841] TS 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313): tablet 15cab4df3cda4e8885f151832b810ef8 (table test-table [id=88fa665c51564fd0bc66354c35b94665]) successfully deleted
I20260924 16:00:04.576732 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 23534
I20260924 16:00:04.626688 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:04.608210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.16.233.189:41823)
0924 16:00:04.608591 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:04.608597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:04.609780 (+ 1183us) server_negotiation.cc:407] Connection header received
0924 16:00:04.610062 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:04.610068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:04.610153 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:04.610234 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:04.612564 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.613736 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.615790 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.616226 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.616377 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:04.619632 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:04.619637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:04.619641 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:04.619678 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:04.623367 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:04.624012 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:04.624018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:04.624022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:04.624216 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:04.624659 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:04.624664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:04.624668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:04.624958 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:04.625243 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:04.625763 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 16:00:04.625978 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260924 16:00:04.629865 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:04.608007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.16.233.188:46427)
0924 16:00:04.608447 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:04.608454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:04.612666 (+ 4212us) server_negotiation.cc:407] Connection header received
0924 16:00:04.613104 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:04.613110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:04.613198 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:04.613323 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:04.615502 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.616677 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.618639 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:04.619281 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:04.619423 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:04.623399 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:04.623406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:04.623411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:04.623455 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:04.627004 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:04.627781 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:04.627786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:04.627790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:04.627943 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:04.628417 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:04.628422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:04.628425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:04.628770 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:04.628998 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:04.629282 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 16:00:04.629557 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260924 16:00:05.371421 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.370439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.129:52709)
0924 16:00:05.371051 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.371171 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":327,"threads_started":1}
W20260924 16:00:05.371858 24559 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:36701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
I20260924 16:00:05.372602 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.371849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.130:55297)
0924 16:00:05.372359 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.372461 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":306,"threads_started":1}
W20260924 16:00:05.373231 24700 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:36701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
I20260924 16:00:05.387247 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.386576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.131:39225)
0924 16:00:05.386964 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.387077 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":202,"threads_started":1}
W20260924 16:00:05.387676 24843 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:36701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
I20260924 16:00:05.883383 24920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4167211be21948888b84c382f610a8b4)
I20260924 16:00:05.883641 24920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } }
I20260924 16:00:05.884701 24920 leader_election.cc:288] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4167211be21948888b84c382f610a8b4 (127.16.233.190:36701), 1c5044a027d640be8f94f8f5bc5ee50a (127.16.233.188:46427)
I20260924 16:00:05.885272 24921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4167211be21948888b84c382f610a8b4)
I20260924 16:00:05.885522 24921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } }
I20260924 16:00:05.886286 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.884858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54558)
0924 16:00:05.885865 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.886032 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":105,"thread_start_us":646,"threads_started":1}
I20260924 16:00:05.886622 24921 leader_election.cc:288] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4167211be21948888b84c382f610a8b4 (127.16.233.190:36701), 85ffe92f9a0e479c940899f0a207cec3 (127.16.233.189:41823)
W20260924 16:00:05.886618 24156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
I20260924 16:00:05.896116 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.884850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:46427 (local address 127.0.0.1:40084)
0924 16:00:05.885464 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.885485 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:05.885667 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:05.886649 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:05.886654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:05.886687 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:05.887042 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:05.887060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.889381 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.889391 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:05.892420 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:05.892448 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.894856 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.894867 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:05.895166 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:05.895232 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 16:00:05.895324 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 16:00:05.895553 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":324,"threads_started":1}
I20260924 16:00:05.897915 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.897506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54572)
0924 16:00:05.897663 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.897763 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 16:00:05.898051 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.886769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54562)
0924 16:00:05.897466 (+ 10697us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.897714 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10508,"thread_start_us":10877,"threads_started":1}
W20260924 16:00:05.898154 24156 leader_election.cc:334] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4167211be21948888b84c382f610a8b4 (127.16.233.190:36701): Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
I20260924 16:00:05.897504 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.885504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.16.233.188:46427)
0924 16:00:05.886218 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:05.886222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:05.886239 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:00:05.886347 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:05.886354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:05.886462 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:05.887240 (+ 778us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:05.887532 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.889169 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.892771 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.894589 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.896726 (+ 2137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:05.896820 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:05.896885 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 16:00:05.897060 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":469,"threads_started":1}
W20260924 16:00:05.898468 24352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
I20260924 16:00:05.899801 24379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85ffe92f9a0e479c940899f0a207cec3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" is_pre_election: true
I20260924 16:00:05.900195 24379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85ffe92f9a0e479c940899f0a207cec3 in term 1.
I20260924 16:00:05.900739 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.900302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54576)
0924 16:00:05.900518 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.900629 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 16:00:05.900724 24158 leader_election.cc:302] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c5044a027d640be8f94f8f5bc5ee50a, 85ffe92f9a0e479c940899f0a207cec3; no voters: 4167211be21948888b84c382f610a8b4
W20260924 16:00:05.900962 24352 leader_election.cc:334] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4167211be21948888b84c382f610a8b4 (127.16.233.190:36701): Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
I20260924 16:00:05.901103 24920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 16:00:05.901237 24920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4167211be21948888b84c382f610a8b4)
I20260924 16:00:05.901374 24920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:00:05.904546 24920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } }
I20260924 16:00:05.905985 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.905466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54598)
0924 16:00:05.905695 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.905790 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260924 16:00:05.906534 24920 leader_election.cc:288] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [CANDIDATE]: Term 2 election: Requested vote from peers 4167211be21948888b84c382f610a8b4 (127.16.233.190:36701), 1c5044a027d640be8f94f8f5bc5ee50a (127.16.233.188:46427)
I20260924 16:00:05.906996 24379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85ffe92f9a0e479c940899f0a207cec3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c5044a027d640be8f94f8f5bc5ee50a"
I20260924 16:00:05.907277 24379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:00:05.907346 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.906886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54608)
0924 16:00:05.907087 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.907164 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 16:00:05.907481 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.887727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.16.233.189:41823)
0924 16:00:05.898422 (+ 10695us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:05.898428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:05.898966 (+ 538us) server_negotiation.cc:407] Connection header received
0924 16:00:05.899188 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:05.899194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:05.899274 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:05.899400 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:05.900147 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.901305 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.903819 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.905952 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.906126 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:05.906221 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:05.906915 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 16:00:05.907067 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10547,"thread_start_us":10371,"threads_started":1}
W20260924 16:00:05.907567 24156 leader_election.cc:334] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4167211be21948888b84c382f610a8b4 (127.16.233.190:36701): Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
I20260924 16:00:05.907255 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.887221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41823 (local address 127.0.0.1:51638)
0924 16:00:05.898847 (+ 11626us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.898862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:05.899026 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:05.899446 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:05.899452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:05.899485 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:05.899915 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:05.899930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.901482 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.901488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:05.903648 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:05.903665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.906266 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.906272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:05.906476 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:05.906538 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 16:00:05.906650 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:00:05.906811 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11513,"thread_start_us":11360,"threads_started":1}
I20260924 16:00:05.909185 24182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "85ffe92f9a0e479c940899f0a207cec3" is_pre_election: true
I20260924 16:00:05.909477 24182 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c5044a027d640be8f94f8f5bc5ee50a in current term 2: Already voted for candidate 85ffe92f9a0e479c940899f0a207cec3 in this term.
I20260924 16:00:05.910080 24379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ffe92f9a0e479c940899f0a207cec3 in term 2.
I20260924 16:00:05.910085 24355 leader_election.cc:302] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c5044a027d640be8f94f8f5bc5ee50a; no voters: 4167211be21948888b84c382f610a8b4, 85ffe92f9a0e479c940899f0a207cec3
I20260924 16:00:05.910461 24921 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:00:05.910557 24158 leader_election.cc:302] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c5044a027d640be8f94f8f5bc5ee50a, 85ffe92f9a0e479c940899f0a207cec3; no voters: 4167211be21948888b84c382f610a8b4
I20260924 16:00:05.910878 24920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:00:05.911423 24920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [term 2 LEADER]: Becoming Leader. State: Replica: 85ffe92f9a0e479c940899f0a207cec3, State: Running, Role: LEADER
I20260924 16:00:05.911819 24920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } }
I20260924 16:00:05.913975 24930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85ffe92f9a0e479c940899f0a207cec3. Latest consensus state: current_term: 2 leader_uuid: "85ffe92f9a0e479c940899f0a207cec3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:05.914435 24930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:05.915110 24932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:00:05.918843 24932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:00:05.919677 24932 catalog_manager.cc:1259] Loaded cluster ID: 13b7d706a32a44ed856c10eab84e6939
I20260924 16:00:05.919750 24932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:00:05.920851 24932 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:00:05.921610 24932 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Loaded TSK: 0
I20260924 16:00:05.922473 24932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:00:05.963655 24172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:00:05.971436 24172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:00:05.975791 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.975358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54626)
0924 16:00:05.975563 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.975641 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 16:00:05.975834 24379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [term 2 FOLLOWER]: Refusing update from remote peer 85ffe92f9a0e479c940899f0a207cec3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 16:00:05.976581 24930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 16:00:05.976802 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.976477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54630)
0924 16:00:05.976616 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.976686 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:00:05.976987 24156 consensus_peers.cc:487] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 -> Peer 4167211be21948888b84c382f610a8b4 (127.16.233.190:36701): Couldn't send request to peer 4167211be21948888b84c382f610a8b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 16:00:05.979199 24921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85ffe92f9a0e479c940899f0a207cec3. Latest consensus state: current_term: 2 leader_uuid: "85ffe92f9a0e479c940899f0a207cec3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:05.979527 24921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:00:05.981179 24930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "85ffe92f9a0e479c940899f0a207cec3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:05.981491 24930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:05.982076 24934 mvcc.cc:205] Tried to move back new op lower bound from 7332927922069831680 to 7332927921819164672. Current Snapshot: MvccSnapshot[applied={T|T < 7332927922069831680}]
I20260924 16:00:05.981974 24921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "85ffe92f9a0e479c940899f0a207cec3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4167211be21948888b84c382f610a8b4" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36701 } } peers { permanent_uuid: "85ffe92f9a0e479c940899f0a207cec3" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "1c5044a027d640be8f94f8f5bc5ee50a" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 46427 } attrs { promote: false } } }
I20260924 16:00:05.982260 24921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5044a027d640be8f94f8f5bc5ee50a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:00:06.002457 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.990881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:45151 (local address 127.0.0.1:47744)
0924 16:00:05.991082 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.991104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:05.991240 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:05.992290 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:05.992298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:05.992333 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:05.992807 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:05.992822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.994929 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.994938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:05.997954 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:05.997976 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.001303 (+ 3327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.001314 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:06.001620 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.001693 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 16:00:06.001788 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:00:06.002014 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 16:00:06.004104 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.991278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.16.233.131:45151)
0924 16:00:05.991859 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:05.991868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:05.991891 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:00:05.991982 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:05.991990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:05.992095 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:05.992984 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:05.993206 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:05.994728 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:05.999130 (+ 4402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.000945 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.003282 (+ 2337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.003400 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:06.003470 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 16:00:06.003742 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":268,"threads_started":1}
I20260924 16:00:06.004926 24775 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce572ea7842413f96bd0a06ee209bb8 (DEFAULT_TABLE table=test-table [id=b91dd36f871847a798fbd4ebc2d00ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:06.005483 24775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce572ea7842413f96bd0a06ee209bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:06.011722 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.992540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:46313 (local address 127.0.0.1:54922)
0924 16:00:05.996734 (+ 4194us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.996760 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:05.996899 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:06.004856 (+ 7957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:06.004864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:06.004907 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:06.005295 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:06.005304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.007012 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.007017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:06.009155 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:06.009168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.010788 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.010799 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:06.010980 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.011019 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 16:00:06.011116 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:00:06.011249 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4006,"mutex_wait_us":45,"thread_start_us":3550,"threads_started":1}
I20260924 16:00:06.011762 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.993046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.16.233.130:46313)
0924 16:00:06.004317 (+ 11271us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:06.004325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:06.004347 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:00:06.004425 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:06.004429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:06.004506 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:06.004751 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:06.005498 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.006660 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.009310 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.010553 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.010727 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.010815 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:06.011188 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 16:00:06.011338 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11140,"thread_start_us":11541,"threads_started":1}
I20260924 16:00:06.013002 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.990277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:39563 (local address 127.0.0.1:50756)
0924 16:00:05.990530 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:05.990549 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:05.990670 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:06.005626 (+ 14956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:06.005633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:06.005671 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:06.005865 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:06.005875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.007266 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.007271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:06.009455 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:06.009471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.012275 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.012281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:06.012452 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.012475 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:00:06.012587 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:00:06.012719 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":133}
I20260924 16:00:06.013046 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:05.990391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.16.233.129:39563)
0924 16:00:06.005276 (+ 14885us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:06.005283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:06.005305 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:00:06.005379 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:06.005383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:06.005469 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:06.005582 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:06.006039 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.007087 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.009621 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.012100 (+ 2479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.012268 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.012339 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:06.012611 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 16:00:06.012761 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14759,"thread_start_us":14660,"threads_started":1}
I20260924 16:00:06.013180 24634 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce572ea7842413f96bd0a06ee209bb8 (DEFAULT_TABLE table=test-table [id=b91dd36f871847a798fbd4ebc2d00ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:06.013767 24634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce572ea7842413f96bd0a06ee209bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:06.013815 24491 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce572ea7842413f96bd0a06ee209bb8 (DEFAULT_TABLE table=test-table [id=b91dd36f871847a798fbd4ebc2d00ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:06.014158 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce572ea7842413f96bd0a06ee209bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:06.019218 24944 tablet_bootstrap.cc:492] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: Bootstrap starting.
I20260924 16:00:06.019865 24945 tablet_bootstrap.cc:492] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: Bootstrap starting.
I20260924 16:00:06.021497 24944 tablet_bootstrap.cc:654] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:06.021677 24945 tablet_bootstrap.cc:654] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:06.023633 24944 tablet_bootstrap.cc:492] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: No bootstrap required, opened a new log
I20260924 16:00:06.023808 24944 ts_tablet_manager.cc:1397] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260924 16:00:06.024480 24944 raft_consensus.cc:348] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } }
I20260924 16:00:06.024695 24944 raft_consensus.cc:374] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:06.024812 24944 raft_consensus.cc:729] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6e7c31a45049689e8e72ade7a08763, State: Initialized, Role: FOLLOWER
I20260924 16:00:06.025089 24944 consensus_queue.cc:260] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } }
I20260924 16:00:06.025863 24944 ts_tablet_manager.cc:1428] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 16:00:06.029750 24844 heartbeater.cc:503] Master 127.16.233.189:41823 was elected leader, sending a full tablet report...
I20260924 16:00:06.030395 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.026480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.131:55213)
0924 16:00:06.029933 (+ 3453us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.030093 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3310,"threads_started":1}
I20260924 16:00:06.030961 24947 tablet_bootstrap.cc:492] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: Bootstrap starting.
I20260924 16:00:06.031705 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.031174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.131:43283)
0924 16:00:06.031453 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.031536 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260924 16:00:06.032706 24945 tablet_bootstrap.cc:492] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: No bootstrap required, opened a new log
I20260924 16:00:06.032891 24945 ts_tablet_manager.cc:1397] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260924 16:00:06.032949 24947 tablet_bootstrap.cc:654] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:06.033615 24945 raft_consensus.cc:348] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } }
I20260924 16:00:06.033829 24945 raft_consensus.cc:374] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:06.033926 24945 raft_consensus.cc:729] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f72561ef010464faf6a7ae00ab1ec7a, State: Initialized, Role: FOLLOWER
I20260924 16:00:06.034189 24945 consensus_queue.cc:260] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } }
I20260924 16:00:06.043063 24945 ts_tablet_manager.cc:1428] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260924 16:00:06.045076 24947 tablet_bootstrap.cc:492] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: No bootstrap required, opened a new log
I20260924 16:00:06.045257 24947 ts_tablet_manager.cc:1397] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.009s
I20260924 16:00:06.045265 24560 heartbeater.cc:503] Master 127.16.233.189:41823 was elected leader, sending a full tablet report...
I20260924 16:00:06.046043 24947 raft_consensus.cc:348] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } }
I20260924 16:00:06.046361 24947 raft_consensus.cc:374] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:06.046471 24947 raft_consensus.cc:729] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef64d9eec314c388971bcb822442cdb, State: Initialized, Role: FOLLOWER
I20260924 16:00:06.046725 24947 consensus_queue.cc:260] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } }
I20260924 16:00:06.049612 24947 ts_tablet_manager.cc:1428] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 16:00:06.050549 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.042938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.129:42719)
0924 16:00:06.050138 (+ 7200us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.050297 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7062,"thread_start_us":6905,"threads_started":1}
I20260924 16:00:06.052120 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.051718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.129:46891)
0924 16:00:06.051917 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.051993 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 16:00:06.052460 24701 heartbeater.cc:503] Master 127.16.233.189:41823 was elected leader, sending a full tablet report...
I20260924 16:00:06.053463 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.050472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.130:46917)
0924 16:00:06.053033 (+ 2561us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.053203 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2433,"threads_started":1}
I20260924 16:00:06.054600 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.054203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.130:48621)
0924 16:00:06.054428 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.054493 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 16:00:06.174806 24952 raft_consensus.cc:482] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:06.175084 24952 raft_consensus.cc:504] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } }
I20260924 16:00:06.175922 24952 leader_election.cc:288] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151), 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313)
I20260924 16:00:06.176606 24654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce572ea7842413f96bd0a06ee209bb8" candidate_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef64d9eec314c388971bcb822442cdb" is_pre_election: true
I20260924 16:00:06.176631 24796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce572ea7842413f96bd0a06ee209bb8" candidate_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6e7c31a45049689e8e72ade7a08763" is_pre_election: true
I20260924 16:00:06.176868 24796 raft_consensus.cc:2454] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f72561ef010464faf6a7ae00ab1ec7a in term 0.
I20260924 16:00:06.176882 24654 raft_consensus.cc:2454] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f72561ef010464faf6a7ae00ab1ec7a in term 0.
I20260924 16:00:06.177377 24443 leader_election.cc:302] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f72561ef010464faf6a7ae00ab1ec7a, ec6e7c31a45049689e8e72ade7a08763; no voters:
I20260924 16:00:06.177656 24952 raft_consensus.cc:2781] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:00:06.177781 24952 raft_consensus.cc:482] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:00:06.177901 24952 raft_consensus.cc:3037] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:06.180061 24952 raft_consensus.cc:504] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } }
I20260924 16:00:06.180660 24952 leader_election.cc:288] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [CANDIDATE]: Term 1 election: Requested vote from peers ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151), 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313)
I20260924 16:00:06.181227 24654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce572ea7842413f96bd0a06ee209bb8" candidate_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef64d9eec314c388971bcb822442cdb"
I20260924 16:00:06.181221 24796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce572ea7842413f96bd0a06ee209bb8" candidate_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6e7c31a45049689e8e72ade7a08763"
I20260924 16:00:06.181447 24654 raft_consensus.cc:3037] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:06.181468 24796 raft_consensus.cc:3037] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:06.183537 24654 raft_consensus.cc:2454] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f72561ef010464faf6a7ae00ab1ec7a in term 1.
I20260924 16:00:06.183573 24796 raft_consensus.cc:2454] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f72561ef010464faf6a7ae00ab1ec7a in term 1.
I20260924 16:00:06.183969 24443 leader_election.cc:302] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef64d9eec314c388971bcb822442cdb, 9f72561ef010464faf6a7ae00ab1ec7a; no voters:
I20260924 16:00:06.184267 24952 raft_consensus.cc:2781] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:06.184720 24952 raft_consensus.cc:686] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 LEADER]: Becoming Leader. State: Replica: 9f72561ef010464faf6a7ae00ab1ec7a, State: Running, Role: LEADER
I20260924 16:00:06.185030 24952 consensus_queue.cc:237] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } }
I20260924 16:00:06.185807 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.185361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.129:45705)
0924 16:00:06.185512 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.185654 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 16:00:06.187999 24171 catalog_manager.cc:5495] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f72561ef010464faf6a7ae00ab1ec7a (127.16.233.129). New cstate: current_term: 1 leader_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f72561ef010464faf6a7ae00ab1ec7a" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 39563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 } health_report { overall_health: UNKNOWN } } }
I20260924 16:00:06.210130 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.204789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.16.233.129:39563)
0924 16:00:06.204981 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:06.204986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:06.205003 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:00:06.205102 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:06.205105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:06.205168 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:06.205233 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:00:06.205653 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.206430 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.208394 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.208898 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.208993 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.209473 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:00:06.209583 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:00:06.209650 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:06.209789 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 16:00:06.209863 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:00:06.213639 24796 raft_consensus.cc:1261] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [term 1 FOLLOWER]: Refusing update from remote peer 9f72561ef010464faf6a7ae00ab1ec7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:06.213668 24654 raft_consensus.cc:1261] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [term 1 FOLLOWER]: Refusing update from remote peer 9f72561ef010464faf6a7ae00ab1ec7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:06.214401 24956 consensus_queue.cc:1035] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6e7c31a45049689e8e72ade7a08763" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 45151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:06.214784 24952 consensus_queue.cc:1035] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef64d9eec314c388971bcb822442cdb" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:06.217850 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.217128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.130:57495)
0924 16:00:06.217430 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.217559 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260924 16:00:06.217931 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.217410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.131:49419)
0924 16:00:06.217655 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.217744 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260924 16:00:06.231290 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.230788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.129:54917)
0924 16:00:06.231030 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.231129 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260924 16:00:06.232331 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.231742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.131:50269)
0924 16:00:06.232031 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.232124 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260924 16:00:06.235461 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.234934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.129:57047)
0924 16:00:06.235138 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.235249 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 16:00:06.235412 24960 mvcc.cc:205] Tried to move back new op lower bound from 7332927923044446208 to 7332927922936586240. Current Snapshot: MvccSnapshot[applied={T|T < 7332927923044446208}]
I20260924 16:00:06.236590 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.236083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.16.233.130:37269)
0924 16:00:06.236332 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.236427 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 16:00:06.240239 24171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:06.240510 24171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:06.246428 24171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85ffe92f9a0e479c940899f0a207cec3: Sending DeleteTablet for 3 replicas of tablet 3ce572ea7842413f96bd0a06ee209bb8
I20260924 16:00:06.247578 24775 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ce572ea7842413f96bd0a06ee209bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:06 UTC) from {username='oryx', principal='oryx/127.16.233.189@KRBTEST.COM'} at 127.0.0.1:47744
I20260924 16:00:06.247581 24491 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ce572ea7842413f96bd0a06ee209bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:06 UTC) from {username='oryx', principal='oryx/127.16.233.189@KRBTEST.COM'} at 127.0.0.1:50756
I20260924 16:00:06.247676 24634 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ce572ea7842413f96bd0a06ee209bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:06 UTC) from {username='oryx', principal='oryx/127.16.233.189@KRBTEST.COM'} at 127.0.0.1:54922
I20260924 16:00:06.248373 24968 tablet_replica.cc:330] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: stopping tablet replica
I20260924 16:00:06.248692 24968 raft_consensus.cc:2229] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:00:06.249337 24968 raft_consensus.cc:2258] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:06.250097 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 24356
I20260924 16:00:06.250981 24968 ts_tablet_manager.cc:1905] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:06.258908 24969 tablet_replica.cc:330] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: stopping tablet replica
I20260924 16:00:06.259212 24969 raft_consensus.cc:2229] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:06.259627 24969 raft_consensus.cc:2258] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:06.261065 24969 ts_tablet_manager.cc:1905] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:06.263360 24970 tablet_replica.cc:330] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: stopping tablet replica
I20260924 16:00:06.263568 24970 raft_consensus.cc:2229] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:06.263893 24970 raft_consensus.cc:2258] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:06.264972 24970 ts_tablet_manager.cc:1905] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:06.268529 24969 ts_tablet_manager.cc:1918] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:06.268760 24969 log.cc:1196] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/3ce572ea7842413f96bd0a06ee209bb8
I20260924 16:00:06.269268 24969 ts_tablet_manager.cc:1939] T 3ce572ea7842413f96bd0a06ee209bb8 P ec6e7c31a45049689e8e72ade7a08763: Deleting consensus metadata
I20260924 16:00:06.269274 24968 ts_tablet_manager.cc:1918] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:06.269418 24968 log.cc:1196] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/3ce572ea7842413f96bd0a06ee209bb8
I20260924 16:00:06.269989 24968 ts_tablet_manager.cc:1939] T 3ce572ea7842413f96bd0a06ee209bb8 P 9f72561ef010464faf6a7ae00ab1ec7a: Deleting consensus metadata
I20260924 16:00:06.270633 24158 catalog_manager.cc:4841] TS ec6e7c31a45049689e8e72ade7a08763 (127.16.233.131:45151): tablet 3ce572ea7842413f96bd0a06ee209bb8 (table test-table [id=b91dd36f871847a798fbd4ebc2d00ded]) successfully deleted
I20260924 16:00:06.271333 24157 catalog_manager.cc:4841] TS 9f72561ef010464faf6a7ae00ab1ec7a (127.16.233.129:39563): tablet 3ce572ea7842413f96bd0a06ee209bb8 (table test-table [id=b91dd36f871847a798fbd4ebc2d00ded]) successfully deleted
I20260924 16:00:06.271737 24970 ts_tablet_manager.cc:1918] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:06.271894 24970 log.cc:1196] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/3ce572ea7842413f96bd0a06ee209bb8
I20260924 16:00:06.272285 24970 ts_tablet_manager.cc:1939] T 3ce572ea7842413f96bd0a06ee209bb8 P 7ef64d9eec314c388971bcb822442cdb: Deleting consensus metadata
I20260924 16:00:06.273178 24158 catalog_manager.cc:4841] TS 7ef64d9eec314c388971bcb822442cdb (127.16.233.130:46313): tablet 3ce572ea7842413f96bd0a06ee209bb8 (table test-table [id=b91dd36f871847a798fbd4ebc2d00ded]) successfully deleted
I20260924 16:00:06.277848 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.251950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.16.233.129:39563)
0924 16:00:06.252176 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:06.252185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:06.262287 (+ 10102us) server_negotiation.cc:407] Connection header received
0924 16:00:06.262730 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:06.262736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:06.262828 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:06.262998 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:06.264618 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.265451 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.266915 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.267484 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.267626 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.271977 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:06.271991 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:06.272007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:06.272063 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:06.275063 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:06.275769 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:06.275776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:06.275779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:06.275923 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:06.276336 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:06.276340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:06.276343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:06.276699 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:06.277017 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:06.277284 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 16:00:06.277502 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:00:06.278627 24552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:06.399683 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 24356
I20260924 16:00:06.423048 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 24570
I20260924 16:00:06.436965 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.423779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.16.233.130:46313)
0924 16:00:06.423990 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:06.423996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:06.424046 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:00:06.424301 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:06.424305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:06.424380 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:06.424491 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:06.425817 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.426592 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.427796 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.428258 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.428391 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.431200 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:06.431224 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:06.431240 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:06.431275 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:06.434180 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:06.434888 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:06.434896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:06.434898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:06.435027 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:06.435415 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:06.435421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:06.435424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:06.435782 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:06.436089 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:06.436396 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 16:00:06.436632 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:00:06.437676 24694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:06.496567 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.496124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54632)
0924 16:00:06.496315 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.496411 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 16:00:06.497870 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.497538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36701 (local address 127.0.0.1:54634)
0924 16:00:06.497684 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:06.497763 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 16:00:06.532745 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 24570
I20260924 16:00:06.550374 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 24712
I20260924 16:00:06.564795 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:06.551151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.16.233.131:45151)
0924 16:00:06.551633 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:06.551637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:06.551653 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:00:06.551708 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:06.551711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:06.551783 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:06.551876 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:06.553177 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.553873 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.555156 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:06.555584 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:06.555681 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:06.558372 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:06.558396 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:06.558419 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:06.558474 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:06.561818 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:06.562748 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:06.562755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:06.562758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:06.562885 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:06.563336 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:06.563341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:06.563343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:06.563721 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:06.563972 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:06.564245 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 16:00:06.564412 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":279,"threads_started":1}
I20260924 16:00:06.565477 24836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:06.660796 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 24712
I20260924 16:00:06.678403 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 24031
I20260924 16:00:06.679524 24202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:06.804806 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 24031
I20260924 16:00:06.827320 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 24133
I20260924 16:00:06.828434 24399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:06.947449 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 24133
2026-09-24T16:00:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10061 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:00:07 dist-test-slave-xn4h krb5kdc[24981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:00:07 dist-test-slave-xn4h krb5kdc[24981](info): set up 2 sockets
Sep 24 16:00:07 dist-test-slave-xn4h krb5kdc[24981](info): commencing operation
krb5kdc: starting...
W20260924 16:00:08.049279 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:00:08 dist-test-slave-xn4h krb5kdc[24981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:00:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.190.keytab.
Entry for principal oryx/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.190.keytab.
I20260924 16:00:08.278411 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45497
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.190.keytab
--principal=oryx/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:08.602847 24995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:08.603238 24995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:08.603404 24995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:08.610780 24995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:00:08.610946 24995 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:00:08.611049 24995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:08.611171 24995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:08.611274 24995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:00:08.611366 24995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:00:08.621622 24995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:45497
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.190.keytab
--principal=oryx/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:08.623025 24995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:08.624445 24995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:08.636523 25000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:08.636590 25001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:08.636613 25003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:08.637609 24995 server_base.cc:1034] running on GCE node
I20260924 16:00:08.638447 24995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:08.639880 24995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:08.641136 24995 hybrid_clock.cc:648] HybridClock initialized: now 1790265608641070 us; error 39 us; skew 500 ppm
I20260924 16:00:08.641489 24995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:08 dist-test-slave-xn4h krb5kdc[24981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265608, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:08.644657 24995 init.cc:377] Logged in from keytab as oryx/127.16.233.190@KRBTEST.COM (short username oryx)
I20260924 16:00:08.646224 24995 webserver.cc:460] Webserver started at http://127.16.233.190:38327/ using document root <none> and password file <none>
I20260924 16:00:08.646984 24995 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:08.647123 24995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:08.647468 24995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:08.650537 24995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "c8d6f190dd644a329f7152290c4e4379"
format_stamp: "Formatted at 2026-09-24 16:00:08 on dist-test-slave-xn4h"
server_key: "b831d85450e21b43465d4fa8cefe802c"
server_key_iv: "66345c55d7b8e38492313f42edd231fb"
server_key_version: "encryptionkey@0"
I20260924 16:00:08.651360 24995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "c8d6f190dd644a329f7152290c4e4379"
format_stamp: "Formatted at 2026-09-24 16:00:08 on dist-test-slave-xn4h"
server_key: "b831d85450e21b43465d4fa8cefe802c"
server_key_iv: "66345c55d7b8e38492313f42edd231fb"
server_key_version: "encryptionkey@0"
I20260924 16:00:08.656481 24995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:00:08.659860 25010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:08.661190 24995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:00:08.661394 24995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "c8d6f190dd644a329f7152290c4e4379"
format_stamp: "Formatted at 2026-09-24 16:00:08 on dist-test-slave-xn4h"
server_key: "b831d85450e21b43465d4fa8cefe802c"
server_key_iv: "66345c55d7b8e38492313f42edd231fb"
server_key_version: "encryptionkey@0"
I20260924 16:00:08.661620 24995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:08.686772 24995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:08.691097 24995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:08.691406 24995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:08.712208 24995 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:45497
I20260924 16:00:08.712222 25072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:45497 every 8 connection(s)
I20260924 16:00:08.713619 24995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:00:08.717527 25073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:08.720621 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 24995
I20260924 16:00:08.720834 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:00:08.721493 17318 external_mini_cluster.cc:1428] Setting key 921bf27e7ac831696c776582e4d4aa06
I20260924 16:00:08.731288 25073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379: Bootstrap starting.
I20260924 16:00:08.735140 25073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:08.736192 25073 log.cc:824] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:00:08 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:08.739363 25073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379: No bootstrap required, opened a new log
I20260924 16:00:08.746531 25073 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d6f190dd644a329f7152290c4e4379" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45497 } }
I20260924 16:00:08.746912 25073 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:08.747028 25073 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d6f190dd644a329f7152290c4e4379, State: Initialized, Role: FOLLOWER
I20260924 16:00:08.747627 25073 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d6f190dd644a329f7152290c4e4379" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45497 } }
I20260924 16:00:08.747872 25073 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:00:08.748008 25073 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:00:08.748183 25073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:08.750414 25073 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d6f190dd644a329f7152290c4e4379" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45497 } }
I20260924 16:00:08.750866 25073 leader_election.cc:302] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8d6f190dd644a329f7152290c4e4379; no voters:
I20260924 16:00:08.751453 25073 leader_election.cc:288] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:00:08.751616 25078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:08.752291 25078 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [term 1 LEADER]: Becoming Leader. State: Replica: c8d6f190dd644a329f7152290c4e4379, State: Running, Role: LEADER
I20260924 16:00:08.752837 25078 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d6f190dd644a329f7152290c4e4379" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45497 } }
I20260924 16:00:08.753619 25073 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:00:08.755743 25079 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8d6f190dd644a329f7152290c4e4379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d6f190dd644a329f7152290c4e4379" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45497 } } }
I20260924 16:00:08.756101 25079 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:08.758978 25080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d6f190dd644a329f7152290c4e4379. Latest consensus state: current_term: 1 leader_uuid: "c8d6f190dd644a329f7152290c4e4379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d6f190dd644a329f7152290c4e4379" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45497 } } }
I20260924 16:00:08.759343 25080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:08.767249 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:08.723703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.16.233.190:45497)
0924 16:00:08.724534 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:08.724551 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:08.724602 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:00:08.726954 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:08.726992 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:08.727586 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:08.728063 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:08.729375 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:08.731347 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:08.732595 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:08.733743 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:08.733929 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:08.738082 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:08.738090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:08.738105 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:08.738147 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:08.757485 (+ 19338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:08.760289 (+ 2804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:08.760302 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:08.760315 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:08.760609 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:08.761270 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:08.761278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:08.761284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:08.762240 (+ 956us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:08.762630 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:08.764902 (+ 2272us) server_negotiation.cc:299] Negotiation successful
0924 16:00:08.765524 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":341,"threads_started":1}
I20260924 16:00:08.767762 25083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:00:08.773597 25083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:00:08.782748 25083 catalog_manager.cc:1347] Generated new cluster ID: 4a94c84b60ac4c8a97a2e1a055ca6402
I20260924 16:00:08.782884 25083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:00:08.815969 25083 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:00:08.817494 25083 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:00:08.824697 25083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379: Generated new TSK 0
I20260924 16:00:08.825553 25083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab.
Entry for principal oryx/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab.
I20260924 16:00:08.890419 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45497
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab
--principal=oryx/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:09.240134 25100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:09.240438 25100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:09.240536 25100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:09.248070 25100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:09.248200 25100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:09.248306 25100 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:00:09.248391 25100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:00:09.258744 25100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab
--principal=oryx/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.190:45497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:09.260008 25100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:09.261358 25100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:09.274734 25108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:09.274724 25106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:09.275226 25105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:09.275555 25100 server_base.cc:1034] running on GCE node
I20260924 16:00:09.276185 25100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:09.277118 25100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:09.278378 25100 hybrid_clock.cc:648] HybridClock initialized: now 1790265609278272 us; error 36 us; skew 500 ppm
I20260924 16:00:09.278692 25100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:09 dist-test-slave-xn4h krb5kdc[24981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265609, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:09.281903 25100 init.cc:377] Logged in from keytab as oryx/127.16.233.129@KRBTEST.COM (short username oryx)
I20260924 16:00:09.283422 25100 webserver.cc:460] Webserver started at http://127.16.233.129:41189/ using document root <none> and password file <none>
I20260924 16:00:09.284123 25100 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:09.284260 25100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:09.284574 25100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:09.287550 25100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d"
format_stamp: "Formatted at 2026-09-24 16:00:09 on dist-test-slave-xn4h"
server_key: "5f852569fcc50cf09da008f4d05e0ff3"
server_key_iv: "30177a81d28a90dd828b621d34ea618b"
server_key_version: "encryptionkey@0"
I20260924 16:00:09.288421 25100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d"
format_stamp: "Formatted at 2026-09-24 16:00:09 on dist-test-slave-xn4h"
server_key: "5f852569fcc50cf09da008f4d05e0ff3"
server_key_iv: "30177a81d28a90dd828b621d34ea618b"
server_key_version: "encryptionkey@0"
I20260924 16:00:09.293483 25100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:00:09.297096 25115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:09.298421 25100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:09.298586 25100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d"
format_stamp: "Formatted at 2026-09-24 16:00:09 on dist-test-slave-xn4h"
server_key: "5f852569fcc50cf09da008f4d05e0ff3"
server_key_iv: "30177a81d28a90dd828b621d34ea618b"
server_key_version: "encryptionkey@0"
I20260924 16:00:09.298818 25100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:09.331655 25100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:09.335829 25100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:09.336124 25100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:09.338892 25100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:09.339020 25100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:09.339177 25100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:09.339249 25100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:00:09 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265609, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:09.364693 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:09.339738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:46625 (local address 127.16.233.190:45497)
0924 16:00:09.340364 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:09.340369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:09.342049 (+ 1680us) server_negotiation.cc:407] Connection header received
0924 16:00:09.344089 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:09.344099 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:09.344236 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:09.344368 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:09.346115 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.347532 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.350277 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.351139 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.351312 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:09.357564 (+ 6252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:09.357569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:09.357573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:09.357613 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:09.360835 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:09.361884 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:09.361890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:09.361894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:09.362067 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:09.362694 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:09.362700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:09.362704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:09.363001 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:09.363157 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:09.364017 (+ 860us) server_negotiation.cc:299] Negotiation successful
0924 16:00:09.364280 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":403,"threads_started":1}
I20260924 16:00:09.365830 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:09.340517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45497 (local address 127.16.233.129:46625)
0924 16:00:09.341373 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:09.341419 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:09.343781 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:09.344639 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:09.344651 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:09.345192 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:09.345883 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:09.345914 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.347737 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.347744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:09.350039 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:09.350067 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.351786 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.351795 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:09.352076 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:09.352938 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:09.352972 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:09.357220 (+ 4248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:09.361004 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:09.361015 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:09.361035 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:09.361755 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:09.362235 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:09.362244 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:09.362248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:09.362566 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:09.363148 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:09.363157 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:09.363590 (+ 433us) client_negotiation.cc:769] Sending connection context
0924 16:00:09.363884 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 16:00:09.364480 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":375,"threads_started":1}
I20260924 16:00:09.391508 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:09.373433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:40007 (local address 127.16.233.190:45497)
0924 16:00:09.373671 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:09.373680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:09.374732 (+ 1052us) server_negotiation.cc:407] Connection header received
0924 16:00:09.375038 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:09.375042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:09.375145 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:09.375265 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:09.377017 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.377958 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.380132 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.380957 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.381135 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:09.384298 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:09.384306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:09.384313 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:09.384368 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:09.387938 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:09.389082 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:09.389088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:09.389093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:09.389314 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:09.389733 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:09.389741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:09.389744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:09.390137 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:09.390364 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:09.390864 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 16:00:09.391105 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:00:09.391570 25188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:09.373401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45497 (local address 127.16.233.129:40007)
0924 16:00:09.374586 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:09.374613 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:09.374804 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:09.375313 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:09.375320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:09.376174 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:09.376804 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:09.376822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.378216 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.378224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:09.379930 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:09.379955 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.381482 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.381492 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:09.381762 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:09.382604 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:09.382637 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:09.384017 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:09.388106 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:09.388115 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:09.388131 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:09.388869 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:09.389461 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:09.389470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:09.389472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:09.389602 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:09.390395 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:09.390401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:09.390590 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 16:00:09.390751 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 16:00:09.391120 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"thread_start_us":684,"threads_started":1}
I20260924 16:00:09.392856 25100 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:36089
I20260924 16:00:09.392940 25237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:36089 every 8 connection(s)
I20260924 16:00:09.394532 25100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:00:09.396993 25238 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45497
I20260924 16:00:09.397192 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 25100
I20260924 16:00:09.397231 25238 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:09.397401 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:00:09.397955 25238 heartbeater.cc:511] Master 127.16.233.190:45497 requested a full tablet report, sending...
I20260924 16:00:09.398041 17318 external_mini_cluster.cc:1428] Setting key 75af0f43d6ef26dab78a22defa7425d9
I20260924 16:00:09.400343 25028 ts_manager.cc:194] Registered new tserver with Master: 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129:36089)
I20260924 16:00:09.401966 25028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:46625
WARNING: no policy specified for oryx/127.16.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab.
Entry for principal oryx/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab.
I20260924 16:00:09.447393 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45497
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab
--principal=oryx/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:09.790118 25242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:09.790472 25242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:09.790573 25242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:09.797719 25242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:09.797821 25242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:09.797904 25242 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:00:09.797982 25242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:00:09.808092 25242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab
--principal=oryx/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.190:45497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:09.809458 25242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:09.810794 25242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:09.824132 25247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:09.824090 25250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:09.824177 25248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:09.830454 25242 server_base.cc:1034] running on GCE node
I20260924 16:00:09.831067 25242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:09.831951 25242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:09.833220 25242 hybrid_clock.cc:648] HybridClock initialized: now 1790265609833160 us; error 37 us; skew 500 ppm
I20260924 16:00:09.833537 25242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:09 dist-test-slave-xn4h krb5kdc[24981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265609, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:09.836675 25242 init.cc:377] Logged in from keytab as oryx/127.16.233.130@KRBTEST.COM (short username oryx)
I20260924 16:00:09.838197 25242 webserver.cc:460] Webserver started at http://127.16.233.130:41187/ using document root <none> and password file <none>
I20260924 16:00:09.838955 25242 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:09.839089 25242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:09.839468 25242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:09.842366 25242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "331f5220441645189cb22c6d17b05de0"
format_stamp: "Formatted at 2026-09-24 16:00:09 on dist-test-slave-xn4h"
server_key: "4fc78e59c60935c7b2044dfd4b5ed059"
server_key_iv: "4faa4413ac9388843326b15cd7b3054b"
server_key_version: "encryptionkey@0"
I20260924 16:00:09.843204 25242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "331f5220441645189cb22c6d17b05de0"
format_stamp: "Formatted at 2026-09-24 16:00:09 on dist-test-slave-xn4h"
server_key: "4fc78e59c60935c7b2044dfd4b5ed059"
server_key_iv: "4faa4413ac9388843326b15cd7b3054b"
server_key_version: "encryptionkey@0"
I20260924 16:00:09.848433 25242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 16:00:09.852174 25257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:09.853682 25242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:00:09.853853 25242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "331f5220441645189cb22c6d17b05de0"
format_stamp: "Formatted at 2026-09-24 16:00:09 on dist-test-slave-xn4h"
server_key: "4fc78e59c60935c7b2044dfd4b5ed059"
server_key_iv: "4faa4413ac9388843326b15cd7b3054b"
server_key_version: "encryptionkey@0"
I20260924 16:00:09.854161 25242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:09.913121 25242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:09.917802 25242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:09.918102 25242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:09.920950 25242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:09.921065 25242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:09.921223 25242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:09.921298 25242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:00:09 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265609, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:09.948661 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:09.921743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:39847 (local address 127.16.233.190:45497)
0924 16:00:09.922232 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:09.922238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:09.923576 (+ 1338us) server_negotiation.cc:407] Connection header received
0924 16:00:09.926885 (+ 3309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:09.926894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:09.927017 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:09.927149 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:09.930052 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.931111 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.934523 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.935058 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.935208 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:09.940424 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:09.940430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:09.940434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:09.940472 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:09.944580 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:09.945547 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:09.945556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:09.945563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:09.945768 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:09.946424 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:09.946432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:09.946437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:09.946930 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:09.947131 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:09.948041 (+ 910us) server_negotiation.cc:299] Negotiation successful
0924 16:00:09.948272 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
I20260924 16:00:09.949743 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:09.922481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45497 (local address 127.16.233.130:39847)
0924 16:00:09.923379 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:09.923436 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:09.926448 (+ 3012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:09.927490 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:09.927504 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:09.928611 (+ 1107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:09.929783 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:09.929817 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.931341 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.931347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:09.934269 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:09.934292 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.935372 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.935383 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:09.935841 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:09.936864 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:09.936887 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:09.940128 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:09.944800 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:09.944815 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:09.944841 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:09.945378 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:09.945913 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:09.945916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:09.945919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:09.946251 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:09.947091 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:09.947097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:09.947553 (+ 456us) client_negotiation.cc:769] Sending connection context
0924 16:00:09.947935 (+ 382us) client_negotiation.cc:240] Negotiation successful
0924 16:00:09.948557 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":359,"threads_started":1}
I20260924 16:00:09.971153 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:09.956146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:44049 (local address 127.16.233.190:45497)
0924 16:00:09.956309 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:09.956313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:09.957053 (+ 740us) server_negotiation.cc:407] Connection header received
0924 16:00:09.957274 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:09.957277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:09.957366 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:09.957481 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:09.958604 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.959683 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.961021 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.961451 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.961582 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:09.963993 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:09.964003 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:09.964006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:09.964038 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:09.967567 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:09.968669 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:09.968680 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:09.968683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:09.968856 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:09.969404 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:09.969413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:09.969417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:09.969937 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:09.970112 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:09.970556 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 16:00:09.970808 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:00:09.971452 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:09.956335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45497 (local address 127.16.233.130:44049)
0924 16:00:09.956944 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:09.956960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:09.957123 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:09.957505 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:09.957509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:09.958074 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:09.958471 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:09.958486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.959825 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.959834 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:09.960878 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:09.960890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:09.961645 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:09.961654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:09.961823 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:09.962497 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:09.962515 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:09.963780 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:09.967754 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:09.967761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:09.967768 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:09.968461 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:09.969033 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:09.969044 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:09.969050 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:09.969245 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:09.970057 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:09.970062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:09.970257 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 16:00:09.970438 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 16:00:09.970833 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":352,"threads_started":1}
I20260924 16:00:09.976647 25242 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:41455
I20260924 16:00:09.976665 25380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:41455 every 8 connection(s)
I20260924 16:00:09.977875 25242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:00:09.979928 25381 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45497
I20260924 16:00:09.980168 25381 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:09.980815 25381 heartbeater.cc:511] Master 127.16.233.190:45497 requested a full tablet report, sending...
I20260924 16:00:09.982169 25028 ts_manager.cc:194] Registered new tserver with Master: 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455)
I20260924 16:00:09.982239 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 25242
I20260924 16:00:09.982434 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:00:09.982885 17318 external_mini_cluster.cc:1428] Setting key 65eda473ec231fed982e67d76174fa73
I20260924 16:00:09.983148 25028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:39847
WARNING: no policy specified for oryx/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab.
Entry for principal oryx/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab.
I20260924 16:00:10.030797 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45497
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab
--principal=oryx/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:10.370054 25385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:10.370455 25385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:10.370648 25385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:10.378829 25385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:10.378997 25385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:10.379143 25385 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:00:10.379298 25385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:00:10.389456 25385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab
--principal=oryx/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.190:45497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:10.391052 25385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:10.392789 25385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:00:10.405032 25238 heartbeater.cc:503] Master 127.16.233.190:45497 was elected leader, sending a full tablet report...
W20260924 16:00:10.406833 25391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:10.407240 25390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:10.407334 25393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:10.408766 25385 server_base.cc:1034] running on GCE node
I20260924 16:00:10.409408 25385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:10.410444 25385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:10.411687 25385 hybrid_clock.cc:648] HybridClock initialized: now 1790265610411622 us; error 40 us; skew 500 ppm
I20260924 16:00:10.412024 25385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:10 dist-test-slave-xn4h krb5kdc[24981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265610, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:10.415328 25385 init.cc:377] Logged in from keytab as oryx/127.16.233.131@KRBTEST.COM (short username oryx)
I20260924 16:00:10.416832 25385 webserver.cc:460] Webserver started at http://127.16.233.131:37967/ using document root <none> and password file <none>
I20260924 16:00:10.417568 25385 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:10.417708 25385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:10.418041 25385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:10.421132 25385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "86662c1918724c8ebdba661787c8ba1b"
format_stamp: "Formatted at 2026-09-24 16:00:10 on dist-test-slave-xn4h"
server_key: "979f487cdfe646b15d6ac5f113162988"
server_key_iv: "3e179ae6d7c3186a4e3fdaa691d37f54"
server_key_version: "encryptionkey@0"
I20260924 16:00:10.422034 25385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "86662c1918724c8ebdba661787c8ba1b"
format_stamp: "Formatted at 2026-09-24 16:00:10 on dist-test-slave-xn4h"
server_key: "979f487cdfe646b15d6ac5f113162988"
server_key_iv: "3e179ae6d7c3186a4e3fdaa691d37f54"
server_key_version: "encryptionkey@0"
I20260924 16:00:10.427371 25385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260924 16:00:10.430960 25400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:10.432363 25385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:00:10.432542 25385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "86662c1918724c8ebdba661787c8ba1b"
format_stamp: "Formatted at 2026-09-24 16:00:10 on dist-test-slave-xn4h"
server_key: "979f487cdfe646b15d6ac5f113162988"
server_key_iv: "3e179ae6d7c3186a4e3fdaa691d37f54"
server_key_version: "encryptionkey@0"
I20260924 16:00:10.432780 25385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:10.451004 25385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:10.455637 25385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:10.455919 25385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:10.458784 25385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:10.458891 25385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:10.459019 25385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:10.459093 25385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:00:10 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265610, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:10.483867 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.459441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:52171 (local address 127.16.233.190:45497)
0924 16:00:10.459643 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.459652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.461146 (+ 1494us) server_negotiation.cc:407] Connection header received
0924 16:00:10.463276 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.463282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.463381 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.463494 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:10.465844 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.467027 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.470371 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.471015 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.471169 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.476865 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:10.476870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:10.476874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:10.476912 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:10.480097 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:10.480856 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:10.480861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:10.480867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:10.481006 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:10.481472 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:10.481478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:10.481481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:10.481785 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:10.481939 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:10.482995 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0924 16:00:10.483335 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:00:10.484598 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.460163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45497 (local address 127.16.233.131:52171)
0924 16:00:10.460983 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:10.461022 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:10.462975 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:10.463749 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:10.463757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:10.464517 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:10.465656 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.465689 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.467229 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.467236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.470139 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.470159 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.471281 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.471290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.471662 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.472579 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:10.472623 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:10.476506 (+ 3883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:10.480263 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:10.480272 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:10.480289 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:10.480724 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:10.481131 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:10.481136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:10.481139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:10.481353 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:10.481932 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:10.481940 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:10.482483 (+ 543us) client_negotiation.cc:769] Sending connection context
0924 16:00:10.482881 (+ 398us) client_negotiation.cc:240] Negotiation successful
0924 16:00:10.483382 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":382,"threads_started":1}
I20260924 16:00:10.509205 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.490834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:39635 (local address 127.16.233.190:45497)
0924 16:00:10.491378 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.491384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.491696 (+ 312us) server_negotiation.cc:407] Connection header received
0924 16:00:10.491936 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.491945 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.492054 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.492172 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:10.493720 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.495028 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.497662 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.498435 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.498581 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.501270 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:10.501278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:10.501286 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:10.501332 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:10.505545 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:10.506645 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:10.506653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:10.506660 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:10.506862 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:10.507330 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:10.507335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:10.507339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:10.507729 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:10.507917 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:10.508476 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 16:00:10.508763 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375}
I20260924 16:00:10.509366 25385 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:33735
I20260924 16:00:10.509415 25471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.490996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45497 (local address 127.16.233.131:39635)
0924 16:00:10.491574 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:10.491593 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:10.491785 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:10.492234 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:10.492241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:10.492999 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:10.493519 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.493539 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.495235 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.495244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.497494 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.497513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.498719 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.498733 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.499000 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.499781 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:10.499813 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:10.501028 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:10.505709 (+ 4681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:10.505716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:10.505722 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:10.506491 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:10.507023 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:10.507033 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:10.507038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:10.507217 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:10.507880 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:10.507888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:10.508132 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 16:00:10.508310 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 16:00:10.508756 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":402,"threads_started":1}
I20260924 16:00:10.509384 25521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:33735 every 8 connection(s)
I20260924 16:00:10.510934 25385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:00:10.512979 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 25385
I20260924 16:00:10.513162 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:00:10.513612 17318 external_mini_cluster.cc:1428] Setting key bdb56256f5cc6c9b7740efdb393c03a2
I20260924 16:00:10.513080 25522 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45497
I20260924 16:00:10.514397 25522 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:10.515312 25522 heartbeater.cc:511] Master 127.16.233.190:45497 requested a full tablet report, sending...
I20260924 16:00:10.516765 25028 ts_manager.cc:194] Registered new tserver with Master: 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735)
I20260924 16:00:10.517572 25028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:52171
I20260924 16:00:10.528959 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:00:10 dist-test-slave-xn4h krb5kdc[24981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:00:10 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:10 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:00:10.579082 25266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:10.568781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.16.233.190:45497)
0924 16:00:10.568977 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.568982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.569401 (+ 419us) server_negotiation.cc:407] Connection header received
0924 16:00:10.569553 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.569557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.569626 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.569771 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:10.571146 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.571882 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.573080 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.573479 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.573569 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.578820 (+ 5251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37138: BlockingRecv error: recv got EOF from 127.0.0.1:37138 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 16:00:10 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:10.595994 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.582490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.16.233.190:45497)
0924 16:00:10.582714 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.582722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.583052 (+ 330us) server_negotiation.cc:407] Connection header received
0924 16:00:10.583258 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.583264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.583339 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.583447 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:10.584618 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.585383 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.586776 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.587278 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.587417 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.590989 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:10.590994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:10.590998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:10.591032 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:10.593343 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:10.593961 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:10.593969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:10.593973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:10.594124 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:10.594576 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:10.594580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:10.594582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:10.594859 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:10.595137 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:10.595453 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 16:00:10.595685 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:00:10.600042 25028 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:00:10.607131 25028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:00:10.629446 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.619053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41455 (local address 127.0.0.1:38496)
0924 16:00:10.620776 (+ 1723us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:10.620790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:10.620928 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:10.621535 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:10.621544 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:10.621571 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:10.622094 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.622109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.623987 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.623993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.626470 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.626486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.628279 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.628286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.628602 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.628663 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 16:00:10.628802 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:00:10.628979 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1625,"mutex_wait_us":47,"thread_start_us":1488,"threads_started":1}
I20260924 16:00:10.630005 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.619936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.16.233.130:41455)
0924 16:00:10.620523 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.620532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.620885 (+ 353us) server_negotiation.cc:407] Connection header received
0924 16:00:10.621139 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.621154 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.621344 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.621528 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:10.622291 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.623822 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.626658 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.628053 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.628226 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.628398 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:10.629320 (+ 922us) server_negotiation.cc:299] Negotiation successful
0924 16:00:10.629518 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":296,"threads_started":1}
I20260924 16:00:10.631124 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.618799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33735 (local address 127.0.0.1:40908)
0924 16:00:10.619586 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:10.619626 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:10.619802 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:10.623494 (+ 3692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:10.623499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:10.623516 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:10.623918 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.623932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.626274 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.626281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.628601 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.628614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.630261 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.630268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.630465 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.630502 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:00:10.630579 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:00:10.630723 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":72,"thread_start_us":389,"threads_started":1}
I20260924 16:00:10.631889 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.619461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:55260)
0924 16:00:10.623240 (+ 3779us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:10.623260 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:10.623361 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:10.625100 (+ 1739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:10.625105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:10.625133 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:10.625537 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.625547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.627213 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.627218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.629095 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.629113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.631237 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.631246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.631414 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.631448 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:00:10.631532 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 16:00:10.631646 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3656,"mutex_wait_us":1,"thread_start_us":3545,"threads_started":1}
I20260924 16:00:10.632218 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.620091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.16.233.131:33735)
0924 16:00:10.622341 (+ 2250us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.622349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.622383 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:00:10.622485 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.622491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.622719 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.624151 (+ 1432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:10.624489 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.626105 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.628773 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.630028 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.631477 (+ 1449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.631645 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:10.631710 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 16:00:10.631893 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2043,"thread_start_us":1793,"threads_started":1}
I20260924 16:00:10.632598 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.619746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.16.233.129:36089)
0924 16:00:10.624663 (+ 4917us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.624668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.624691 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:00:10.624766 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.624771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.624919 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.625120 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:10.625701 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.627044 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.629280 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.630738 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.630907 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.631078 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:10.632176 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0924 16:00:10.632357 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4778,"thread_start_us":4638,"threads_started":1}
I20260924 16:00:10.634379 25310 tablet_service.cc:1467] Processing CreateTablet for tablet c4f833b48d9e44f8bd6b5f62a49d039f (DEFAULT_TABLE table=test-table [id=4a398ed6ac754d558ed5fca467a8c3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:10.635952 25310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f833b48d9e44f8bd6b5f62a49d039f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:10.635756 25451 tablet_service.cc:1467] Processing CreateTablet for tablet c4f833b48d9e44f8bd6b5f62a49d039f (DEFAULT_TABLE table=test-table [id=4a398ed6ac754d558ed5fca467a8c3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:10.636029 25167 tablet_service.cc:1467] Processing CreateTablet for tablet c4f833b48d9e44f8bd6b5f62a49d039f (DEFAULT_TABLE table=test-table [id=4a398ed6ac754d558ed5fca467a8c3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:10.637111 25451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f833b48d9e44f8bd6b5f62a49d039f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:10.637388 25167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f833b48d9e44f8bd6b5f62a49d039f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:10.648453 25546 tablet_bootstrap.cc:492] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: Bootstrap starting.
I20260924 16:00:10.650684 25547 tablet_bootstrap.cc:492] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: Bootstrap starting.
I20260924 16:00:10.652690 25546 tablet_bootstrap.cc:654] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:10.654140 25547 tablet_bootstrap.cc:654] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:10.654925 25546 log.cc:824] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:10.655019 25548 tablet_bootstrap.cc:492] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: Bootstrap starting.
I20260924 16:00:10.655422 25547 log.cc:824] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:10.658906 25548 tablet_bootstrap.cc:654] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:10.659040 25546 tablet_bootstrap.cc:492] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: No bootstrap required, opened a new log
I20260924 16:00:10.659257 25547 tablet_bootstrap.cc:492] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: No bootstrap required, opened a new log
I20260924 16:00:10.659301 25546 ts_tablet_manager.cc:1397] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 16:00:10.659531 25547 ts_tablet_manager.cc:1397] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:00:10.660200 25548 log.cc:824] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:10.663467 25548 tablet_bootstrap.cc:492] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: No bootstrap required, opened a new log
I20260924 16:00:10.663746 25548 ts_tablet_manager.cc:1397] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:00:10.667897 25546 raft_consensus.cc:348] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } }
I20260924 16:00:10.668298 25546 raft_consensus.cc:374] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:10.668430 25546 raft_consensus.cc:729] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86662c1918724c8ebdba661787c8ba1b, State: Initialized, Role: FOLLOWER
I20260924 16:00:10.669149 25546 consensus_queue.cc:260] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } }
I20260924 16:00:10.670382 25522 heartbeater.cc:503] Master 127.16.233.190:45497 was elected leader, sending a full tablet report...
I20260924 16:00:10.670106 25547 raft_consensus.cc:348] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } }
I20260924 16:00:10.670634 25547 raft_consensus.cc:374] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:10.670737 25547 raft_consensus.cc:729] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331f5220441645189cb22c6d17b05de0, State: Initialized, Role: FOLLOWER
I20260924 16:00:10.671334 25547 consensus_queue.cc:260] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } }
I20260924 16:00:10.672770 25546 ts_tablet_manager.cc:1428] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260924 16:00:10.675746 25548 raft_consensus.cc:348] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } }
I20260924 16:00:10.676066 25381 heartbeater.cc:503] Master 127.16.233.190:45497 was elected leader, sending a full tablet report...
I20260924 16:00:10.676222 25548 raft_consensus.cc:374] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:10.676326 25548 raft_consensus.cc:729] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bfb4a627ad4fe5ae9b3e32ce60831d, State: Initialized, Role: FOLLOWER
I20260924 16:00:10.676766 25547 ts_tablet_manager.cc:1428] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260924 16:00:10.677021 25548 consensus_queue.cc:260] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } }
I20260924 16:00:10.678830 25548 ts_tablet_manager.cc:1428] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260924 16:00:10.699699 25553 raft_consensus.cc:482] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:10.700062 25553 raft_consensus.cc:504] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } }
I20260924 16:00:10.701774 25553 leader_election.cc:288] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735), 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129:36089)
I20260924 16:00:10.709018 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.702105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:35477 (local address 127.16.233.131:33735)
0924 16:00:10.702334 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.702340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.703308 (+ 968us) server_negotiation.cc:407] Connection header received
0924 16:00:10.703582 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.703588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.703667 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.703761 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:10.704541 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.705257 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.707323 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.708062 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.708246 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.708333 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:10.708629 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 16:00:10.708760 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:00:10.709035 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.702211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:36457 (local address 127.16.233.129:36089)
0924 16:00:10.702448 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.702454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.703423 (+ 969us) server_negotiation.cc:407] Connection header received
0924 16:00:10.703677 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.703681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.703773 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.703851 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:10.704689 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.705385 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.707310 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.708068 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.708267 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.708358 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:10.708674 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 16:00:10.708797 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:00:10.709172 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.702222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33735 (local address 127.16.233.130:35477)
0924 16:00:10.703179 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:10.703199 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:10.703411 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:10.703794 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:10.703798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:10.703822 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:10.704376 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.704390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.705382 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.705390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.707155 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.707166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.708306 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.708314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.708472 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.708495 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:00:10.708601 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:00:10.708724 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":100,"thread_start_us":448,"threads_started":1}
I20260924 16:00:10.709194 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.702369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.16.233.130:36457)
0924 16:00:10.703335 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:10.703354 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:10.703478 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:10.703969 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:10.703974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:10.704008 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:10.704539 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.704549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.705527 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.705532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.707153 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:10.707165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.708363 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.708370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:10.708520 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.708542 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:00:10.708646 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:00:10.708763 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":27,"thread_start_us":596,"threads_started":1}
I20260924 16:00:10.710243 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f833b48d9e44f8bd6b5f62a49d039f" candidate_uuid: "331f5220441645189cb22c6d17b05de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b" is_pre_election: true
I20260924 16:00:10.710275 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f833b48d9e44f8bd6b5f62a49d039f" candidate_uuid: "331f5220441645189cb22c6d17b05de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" is_pre_election: true
I20260924 16:00:10.710800 25193 raft_consensus.cc:2454] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 331f5220441645189cb22c6d17b05de0 in term 0.
I20260924 16:00:10.710800 25477 raft_consensus.cc:2454] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 331f5220441645189cb22c6d17b05de0 in term 0.
I20260924 16:00:10.711421 25260 leader_election.cc:302] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bfb4a627ad4fe5ae9b3e32ce60831d, 331f5220441645189cb22c6d17b05de0; no voters:
I20260924 16:00:10.711827 25553 raft_consensus.cc:2781] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:00:10.711970 25553 raft_consensus.cc:482] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:00:10.712095 25553 raft_consensus.cc:3037] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:10.714233 25553 raft_consensus.cc:504] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } }
I20260924 16:00:10.714879 25553 leader_election.cc:288] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [CANDIDATE]: Term 1 election: Requested vote from peers 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735), 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129:36089)
I20260924 16:00:10.715302 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f833b48d9e44f8bd6b5f62a49d039f" candidate_uuid: "331f5220441645189cb22c6d17b05de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b"
I20260924 16:00:10.715411 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f833b48d9e44f8bd6b5f62a49d039f" candidate_uuid: "331f5220441645189cb22c6d17b05de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d"
I20260924 16:00:10.715495 25477 raft_consensus.cc:3037] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:10.715615 25193 raft_consensus.cc:3037] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:10.717715 25477 raft_consensus.cc:2454] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 331f5220441645189cb22c6d17b05de0 in term 1.
I20260924 16:00:10.718091 25261 leader_election.cc:302] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 331f5220441645189cb22c6d17b05de0, 86662c1918724c8ebdba661787c8ba1b; no voters:
I20260924 16:00:10.718387 25553 raft_consensus.cc:2781] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:10.718726 25193 raft_consensus.cc:2454] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 331f5220441645189cb22c6d17b05de0 in term 1.
I20260924 16:00:10.718917 25553 raft_consensus.cc:686] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 1 LEADER]: Becoming Leader. State: Replica: 331f5220441645189cb22c6d17b05de0, State: Running, Role: LEADER
I20260924 16:00:10.719507 25553 consensus_queue.cc:237] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } }
I20260924 16:00:10.724328 25028 catalog_manager.cc:5495] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 331f5220441645189cb22c6d17b05de0 (127.16.233.130). New cstate: current_term: 1 leader_uuid: "331f5220441645189cb22c6d17b05de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } health_report { overall_health: UNKNOWN } } }
W20260924 16:00:10.729879 25382 tablet.cc:2369] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:00:10.759616 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:10.754291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.16.233.130:41455)
0924 16:00:10.754553 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:10.754559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:10.754793 (+ 234us) server_negotiation.cc:407] Connection header received
0924 16:00:10.754985 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:10.754989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:10.755075 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:10.755208 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:00:10.755502 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.756187 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.757542 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:10.757975 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:10.758066 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:10.758555 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:00:10.759151 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:00:10.759277 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:10.759345 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 16:00:10.759411 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260924 16:00:10.763142 25523 tablet.cc:2369] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:00:10.768250 25477 raft_consensus.cc:1261] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Refusing update from remote peer 331f5220441645189cb22c6d17b05de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:10.768249 25193 raft_consensus.cc:1261] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Refusing update from remote peer 331f5220441645189cb22c6d17b05de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:10.769380 25553 consensus_queue.cc:1035] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:10.769734 25557 consensus_queue.cc:1035] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:10.779974 25560 mvcc.cc:205] Tried to move back new op lower bound from 7332927941696147456 to 7332927941511016448. Current Snapshot: MvccSnapshot[applied={T|T < 7332927941696147456}]
I20260924 16:00:10.794135 25026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:10.794509 25026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:10.798236 25026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379: Sending DeleteTablet for 3 replicas of tablet c4f833b48d9e44f8bd6b5f62a49d039f
I20260924 16:00:10.799484 25451 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f833b48d9e44f8bd6b5f62a49d039f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:10 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:40908
I20260924 16:00:10.799538 25310 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f833b48d9e44f8bd6b5f62a49d039f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:10 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:38496
I20260924 16:00:10.800282 25167 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f833b48d9e44f8bd6b5f62a49d039f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:10 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:55260
I20260924 16:00:10.801415 25026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:00:10.802675 25026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:00:10.804919 25571 tablet_replica.cc:330] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: stopping tablet replica
I20260924 16:00:10.805565 25570 tablet_replica.cc:330] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: stopping tablet replica
I20260924 16:00:10.806097 25570 raft_consensus.cc:2229] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:10.806623 25571 raft_consensus.cc:2229] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:00:10.807194 25570 raft_consensus.cc:2258] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:10.807945 25571 raft_consensus.cc:2258] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:10.811551 25570 ts_tablet_manager.cc:1905] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:10.814272 25166 tablet_service.cc:1467] Processing CreateTablet for tablet 54b3bd0a0297417b95f3707debeb698e (DEFAULT_TABLE table=test-table [id=1d62bc3118c1417bb5e0d398e8a13e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:10.814782 25572 tablet_replica.cc:330] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: stopping tablet replica
I20260924 16:00:10.814962 25166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3bd0a0297417b95f3707debeb698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:10.815080 25310 tablet_service.cc:1467] Processing CreateTablet for tablet 54b3bd0a0297417b95f3707debeb698e (DEFAULT_TABLE table=test-table [id=1d62bc3118c1417bb5e0d398e8a13e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:10.815207 25572 raft_consensus.cc:2229] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:10.815686 25572 raft_consensus.cc:2258] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:10.815811 25451 tablet_service.cc:1467] Processing CreateTablet for tablet 54b3bd0a0297417b95f3707debeb698e (DEFAULT_TABLE table=test-table [id=1d62bc3118c1417bb5e0d398e8a13e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:10.817523 25451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3bd0a0297417b95f3707debeb698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:10.817737 25572 ts_tablet_manager.cc:1905] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:10.819013 25570 ts_tablet_manager.cc:1918] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:10.819236 25570 log.cc:1196] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/c4f833b48d9e44f8bd6b5f62a49d039f
I20260924 16:00:10.819880 25570 ts_tablet_manager.cc:1939] T c4f833b48d9e44f8bd6b5f62a49d039f P 86662c1918724c8ebdba661787c8ba1b: Deleting consensus metadata
I20260924 16:00:10.821668 25548 tablet_bootstrap.cc:492] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: Bootstrap starting.
I20260924 16:00:10.821770 25014 catalog_manager.cc:4841] TS 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735): tablet c4f833b48d9e44f8bd6b5f62a49d039f (table test-table [id=4a398ed6ac754d558ed5fca467a8c3dc]) successfully deleted
I20260924 16:00:10.822528 25310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3bd0a0297417b95f3707debeb698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:10.823561 25548 tablet_bootstrap.cc:654] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:10.825789 25572 ts_tablet_manager.cc:1918] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:10.826056 25572 log.cc:1196] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/c4f833b48d9e44f8bd6b5f62a49d039f
I20260924 16:00:10.826845 25572 ts_tablet_manager.cc:1939] T c4f833b48d9e44f8bd6b5f62a49d039f P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting consensus metadata
I20260924 16:00:10.826951 25546 tablet_bootstrap.cc:492] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: Bootstrap starting.
I20260924 16:00:10.828498 25546 tablet_bootstrap.cc:654] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:10.829051 25013 catalog_manager.cc:4841] TS 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129:36089): tablet c4f833b48d9e44f8bd6b5f62a49d039f (table test-table [id=4a398ed6ac754d558ed5fca467a8c3dc]) successfully deleted
I20260924 16:00:10.829542 25547 tablet_bootstrap.cc:492] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: Bootstrap starting.
I20260924 16:00:10.830171 25571 ts_tablet_manager.cc:1905] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:10.831794 25547 tablet_bootstrap.cc:654] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:10.832695 25546 tablet_bootstrap.cc:492] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: No bootstrap required, opened a new log
I20260924 16:00:10.832839 25546 ts_tablet_manager.cc:1397] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 16:00:10.833572 25546 raft_consensus.cc:348] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:10.833811 25546 raft_consensus.cc:374] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:10.833907 25546 raft_consensus.cc:729] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86662c1918724c8ebdba661787c8ba1b, State: Initialized, Role: FOLLOWER
I20260924 16:00:10.834153 25546 consensus_queue.cc:260] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:10.834816 25546 ts_tablet_manager.cc:1428] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:00:10.836483 25548 tablet_bootstrap.cc:492] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: No bootstrap required, opened a new log
I20260924 16:00:10.836661 25548 ts_tablet_manager.cc:1397] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260924 16:00:10.837203 25571 ts_tablet_manager.cc:1918] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:10.837363 25571 log.cc:1196] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/c4f833b48d9e44f8bd6b5f62a49d039f
I20260924 16:00:10.837672 25548 raft_consensus.cc:348] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:10.837925 25571 ts_tablet_manager.cc:1939] T c4f833b48d9e44f8bd6b5f62a49d039f P 331f5220441645189cb22c6d17b05de0: Deleting consensus metadata
I20260924 16:00:10.837965 25548 raft_consensus.cc:374] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:10.838074 25548 raft_consensus.cc:729] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bfb4a627ad4fe5ae9b3e32ce60831d, State: Initialized, Role: FOLLOWER
I20260924 16:00:10.838474 25548 consensus_queue.cc:260] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:10.841629 25548 ts_tablet_manager.cc:1428] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260924 16:00:10.842815 25547 tablet_bootstrap.cc:492] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: No bootstrap required, opened a new log
I20260924 16:00:10.843036 25547 ts_tablet_manager.cc:1397] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260924 16:00:10.843366 25011 catalog_manager.cc:4841] TS 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455): tablet c4f833b48d9e44f8bd6b5f62a49d039f (table test-table [id=4a398ed6ac754d558ed5fca467a8c3dc]) successfully deleted
I20260924 16:00:10.843992 25547 raft_consensus.cc:348] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:10.844161 25547 raft_consensus.cc:374] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:10.844288 25547 raft_consensus.cc:729] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331f5220441645189cb22c6d17b05de0, State: Initialized, Role: FOLLOWER
I20260924 16:00:10.844554 25547 consensus_queue.cc:260] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:10.845192 25547 ts_tablet_manager.cc:1428] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260924 16:00:10.898047 25239 tablet.cc:2369] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:00:11.052239 25554 raft_consensus.cc:482] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:11.052500 25554 raft_consensus.cc:504] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.054225 25554 leader_election.cc:288] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455), 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735)
I20260924 16:00:11.063568 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.054937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:59417 (local address 127.16.233.130:41455)
0924 16:00:11.055143 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.055151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.056251 (+ 1100us) server_negotiation.cc:407] Connection header received
0924 16:00:11.056578 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.056584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.056659 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.056755 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:11.057618 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.058901 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.061543 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.062408 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.062592 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.062672 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:11.063099 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 16:00:11.063245 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:00:11.063577 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.054494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41455 (local address 127.16.233.129:59417)
0924 16:00:11.056139 (+ 1645us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:11.056160 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:11.056392 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:11.056783 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:11.056787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:11.056866 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:11.057355 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.057369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.059114 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.059119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.061324 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.061341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.062623 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.062629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.062850 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.062897 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 16:00:11.063040 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 16:00:11.063173 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1491,"mutex_wait_us":12,"thread_start_us":1648,"threads_started":1}
I20260924 16:00:11.063701 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.055008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:43993 (local address 127.16.233.131:33735)
0924 16:00:11.055232 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.055241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.056618 (+ 1377us) server_negotiation.cc:407] Connection header received
0924 16:00:11.056774 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.056781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.056896 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.057002 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:11.057708 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.058716 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.061500 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.062700 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.062878 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.062950 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:11.063307 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 16:00:11.063452 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:00:11.064383 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.054633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:33735 (local address 127.16.233.129:43993)
0924 16:00:11.056496 (+ 1863us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:11.056514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:11.056636 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:11.057047 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:11.057053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:11.057081 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:11.057508 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.057519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.058858 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.058863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.061342 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.061351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.062936 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.062944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.063118 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.063146 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:00:11.063864 (+ 718us) client_negotiation.cc:240] Negotiation successful
0924 16:00:11.063998 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1709,"mutex_wait_us":95,"thread_start_us":1167,"threads_started":1}
I20260924 16:00:11.064919 25330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3bd0a0297417b95f3707debeb698e" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "331f5220441645189cb22c6d17b05de0" is_pre_election: true
I20260924 16:00:11.065104 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3bd0a0297417b95f3707debeb698e" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b" is_pre_election: true
I20260924 16:00:11.065367 25477 raft_consensus.cc:2454] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 0.
I20260924 16:00:11.065433 25330 raft_consensus.cc:2454] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 0.
I20260924 16:00:11.065878 25119 leader_election.cc:302] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bfb4a627ad4fe5ae9b3e32ce60831d, 86662c1918724c8ebdba661787c8ba1b; no voters:
I20260924 16:00:11.066357 25554 raft_consensus.cc:2781] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:00:11.066491 25554 raft_consensus.cc:482] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:00:11.066597 25554 raft_consensus.cc:3037] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.068878 25554 raft_consensus.cc:504] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.069578 25554 leader_election.cc:288] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 election: Requested vote from peers 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455), 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735)
I20260924 16:00:11.070068 25330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3bd0a0297417b95f3707debeb698e" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "331f5220441645189cb22c6d17b05de0"
I20260924 16:00:11.070103 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3bd0a0297417b95f3707debeb698e" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b"
I20260924 16:00:11.070322 25477 raft_consensus.cc:3037] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.070329 25330 raft_consensus.cc:3037] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.073251 25477 raft_consensus.cc:2454] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 1.
I20260924 16:00:11.073578 25330 raft_consensus.cc:2454] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 1.
I20260924 16:00:11.073665 25119 leader_election.cc:302] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bfb4a627ad4fe5ae9b3e32ce60831d, 86662c1918724c8ebdba661787c8ba1b; no voters:
I20260924 16:00:11.073989 25554 raft_consensus.cc:2781] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:11.074527 25554 raft_consensus.cc:686] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 LEADER]: Becoming Leader. State: Replica: 28bfb4a627ad4fe5ae9b3e32ce60831d, State: Running, Role: LEADER
I20260924 16:00:11.075110 25554 consensus_queue.cc:237] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.078842 25027 catalog_manager.cc:5495] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d reported cstate change: term changed from 0 to 1, leader changed from <none> to 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129). New cstate: current_term: 1 leader_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:00:11 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265608, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.190@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:11.151938 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.136911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.16.233.190:45497)
0924 16:00:11.137849 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.137857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.137879 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:00:11.137942 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.137947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.138036 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.138163 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:11.139226 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.140366 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.141559 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.142198 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.142359 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.145479 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:11.145486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:11.145492 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:11.145529 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:11.149097 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:11.149795 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:11.149800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:11.149822 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:11.149986 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:11.150371 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:11.150378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:11.150382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:11.150741 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:11.150925 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:11.151297 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 16:00:11.151516 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":591,"threads_started":1}
I20260924 16:00:11.152149 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.136868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45497 (local address 127.0.0.1:37158)
0924 16:00:11.137494 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:11.137523 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:11.137705 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:11.138186 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:11.138189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:11.138848 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:11.139037 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.139045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.140508 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.140512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.141409 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.141419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.142397 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.142401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.142545 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.143210 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:11.143227 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:11.145249 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:11.149254 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:11.149265 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:11.149270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:11.149659 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:11.150100 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:11.150103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:11.150105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:11.150241 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:11.150874 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:11.150881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:11.151020 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 16:00:11.151211 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 16:00:11.151648 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":143,"thread_start_us":278,"threads_started":1}
I20260924 16:00:11.157919 25027 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:37158:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:00:11.159334 25027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:00:11.166991 25166 tablet_service.cc:1467] Processing CreateTablet for tablet db0d94e1b7f3498ca8e29407f55b913a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=341a2da39da2486090c6f7476b5fe02b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:00:11.167585 25166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0d94e1b7f3498ca8e29407f55b913a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:11.167400 25451 tablet_service.cc:1467] Processing CreateTablet for tablet db0d94e1b7f3498ca8e29407f55b913a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=341a2da39da2486090c6f7476b5fe02b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:00:11.167958 25451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0d94e1b7f3498ca8e29407f55b913a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:11.171277 25310 tablet_service.cc:1467] Processing CreateTablet for tablet db0d94e1b7f3498ca8e29407f55b913a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=341a2da39da2486090c6f7476b5fe02b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:00:11.171864 25310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0d94e1b7f3498ca8e29407f55b913a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:11.174234 25548 tablet_bootstrap.cc:492] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: Bootstrap starting.
I20260924 16:00:11.175155 25546 tablet_bootstrap.cc:492] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: Bootstrap starting.
I20260924 16:00:11.175877 25548 tablet_bootstrap.cc:654] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:11.176838 25546 tablet_bootstrap.cc:654] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:11.180279 25547 tablet_bootstrap.cc:492] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: Bootstrap starting.
I20260924 16:00:11.182065 25547 tablet_bootstrap.cc:654] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:11.188071 25546 tablet_bootstrap.cc:492] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: No bootstrap required, opened a new log
I20260924 16:00:11.188246 25546 ts_tablet_manager.cc:1397] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260924 16:00:11.188256 25547 tablet_bootstrap.cc:492] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: No bootstrap required, opened a new log
I20260924 16:00:11.188432 25547 ts_tablet_manager.cc:1397] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 16:00:11.188480 25548 tablet_bootstrap.cc:492] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: No bootstrap required, opened a new log
I20260924 16:00:11.188629 25548 ts_tablet_manager.cc:1397] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260924 16:00:11.189029 25546 raft_consensus.cc:348] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.189266 25546 raft_consensus.cc:374] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:11.189350 25546 raft_consensus.cc:729] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86662c1918724c8ebdba661787c8ba1b, State: Initialized, Role: FOLLOWER
I20260924 16:00:11.189231 25547 raft_consensus.cc:348] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.189337 25548 raft_consensus.cc:348] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.189571 25548 raft_consensus.cc:374] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:11.189654 25548 raft_consensus.cc:729] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bfb4a627ad4fe5ae9b3e32ce60831d, State: Initialized, Role: FOLLOWER
I20260924 16:00:11.189487 25547 raft_consensus.cc:374] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:11.189779 25547 raft_consensus.cc:729] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331f5220441645189cb22c6d17b05de0, State: Initialized, Role: FOLLOWER
I20260924 16:00:11.189772 25546 consensus_queue.cc:260] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.189927 25548 consensus_queue.cc:260] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.190354 25554 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.190604 25548 ts_tablet_manager.cc:1428] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 16:00:11.190651 25554 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:00:11.190163 25547 consensus_queue.cc:260] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.191401 25546 ts_tablet_manager.cc:1428] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:00:11.191444 25553 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.191730 25553 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:00:11.192739 25552 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.193040 25552 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:00:11.193140 25547 ts_tablet_manager.cc:1428] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 16:00:11.248613 25553 raft_consensus.cc:482] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:11.248855 25553 raft_consensus.cc:504] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.249620 25553 leader_election.cc:288] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129:36089), 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735)
I20260924 16:00:11.250177 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0d94e1b7f3498ca8e29407f55b913a" candidate_uuid: "331f5220441645189cb22c6d17b05de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" is_pre_election: true
I20260924 16:00:11.250172 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0d94e1b7f3498ca8e29407f55b913a" candidate_uuid: "331f5220441645189cb22c6d17b05de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b" is_pre_election: true
I20260924 16:00:11.250453 25193 raft_consensus.cc:2454] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 331f5220441645189cb22c6d17b05de0 in term 0.
I20260924 16:00:11.250456 25477 raft_consensus.cc:2454] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 331f5220441645189cb22c6d17b05de0 in term 0.
I20260924 16:00:11.250903 25261 leader_election.cc:302] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 331f5220441645189cb22c6d17b05de0, 86662c1918724c8ebdba661787c8ba1b; no voters:
I20260924 16:00:11.251173 25553 raft_consensus.cc:2781] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:00:11.251320 25553 raft_consensus.cc:482] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:00:11.251411 25553 raft_consensus.cc:3037] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.254141 25553 raft_consensus.cc:504] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.254870 25553 leader_election.cc:288] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [CANDIDATE]: Term 1 election: Requested vote from peers 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129:36089), 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735)
I20260924 16:00:11.255337 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0d94e1b7f3498ca8e29407f55b913a" candidate_uuid: "331f5220441645189cb22c6d17b05de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d"
I20260924 16:00:11.255426 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0d94e1b7f3498ca8e29407f55b913a" candidate_uuid: "331f5220441645189cb22c6d17b05de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b"
I20260924 16:00:11.255566 25193 raft_consensus.cc:3037] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.255651 25477 raft_consensus.cc:3037] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.257606 25193 raft_consensus.cc:2454] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 331f5220441645189cb22c6d17b05de0 in term 1.
I20260924 16:00:11.257635 25477 raft_consensus.cc:2454] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 331f5220441645189cb22c6d17b05de0 in term 1.
I20260924 16:00:11.258009 25260 leader_election.cc:302] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bfb4a627ad4fe5ae9b3e32ce60831d, 331f5220441645189cb22c6d17b05de0; no voters:
I20260924 16:00:11.258284 25553 raft_consensus.cc:2781] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:11.258476 25553 raft_consensus.cc:686] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [term 1 LEADER]: Becoming Leader. State: Replica: 331f5220441645189cb22c6d17b05de0, State: Running, Role: LEADER
I20260924 16:00:11.258795 25553 consensus_queue.cc:237] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.260255 25566 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: TxnStatusTablet state changed. Reason: New leader 331f5220441645189cb22c6d17b05de0. Latest consensus state: current_term: 1 leader_uuid: "331f5220441645189cb22c6d17b05de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.260540 25566 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:00:11.261190 25592 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:00:11.261355 25592 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:00:11.261677 25028 catalog_manager.cc:5495] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 331f5220441645189cb22c6d17b05de0 (127.16.233.130). New cstate: current_term: 1 leader_uuid: "331f5220441645189cb22c6d17b05de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } health_report { overall_health: UNKNOWN } } }
I20260924 16:00:11.287580 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.281376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.16.233.130:41455)
0924 16:00:11.281541 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.281545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.281869 (+ 324us) server_negotiation.cc:407] Connection header received
0924 16:00:11.282172 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.282177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.282337 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.282428 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:00:11.282828 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.283533 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.285488 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.286036 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.286132 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.286771 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:00:11.286873 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:00:11.286944 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:11.287121 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 16:00:11.287219 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:00:11.287607 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.281432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41455 (local address 127.0.0.1:38504)
0924 16:00:11.281776 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:11.281797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:11.281956 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:11.282471 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:11.282476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:11.282489 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:00:11.282679 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.282691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.283663 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.283670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.285342 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.285357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.286186 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.286195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.286462 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.286537 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:00:11.286977 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:00:11.287001 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:00:11.287091 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 16:00:11.287271 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 16:00:11.291798 25193 raft_consensus.cc:1261] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Refusing update from remote peer 331f5220441645189cb22c6d17b05de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:11.292439 25477 raft_consensus.cc:1261] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Refusing update from remote peer 331f5220441645189cb22c6d17b05de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:11.292414 25566 consensus_queue.cc:1035] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:11.292991 25566 consensus_queue.cc:1035] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:11.296190 25554 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: TxnStatusTablet state changed. Reason: New leader 331f5220441645189cb22c6d17b05de0. Latest consensus state: current_term: 1 leader_uuid: "331f5220441645189cb22c6d17b05de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.296470 25554 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:00:11.299047 25595 mvcc.cc:205] Tried to move back new op lower bound from 7332927943845289984 to 7332927943717883904. Current Snapshot: MvccSnapshot[applied={T|T < 7332927943845289984}]
I20260924 16:00:11.297922 25552 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: TxnStatusTablet state changed. Reason: New leader 331f5220441645189cb22c6d17b05de0. Latest consensus state: current_term: 1 leader_uuid: "331f5220441645189cb22c6d17b05de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.299257 25552 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:00:11.302641 25553 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "331f5220441645189cb22c6d17b05de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.303020 25553 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:00:11.304591 25554 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "331f5220441645189cb22c6d17b05de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.304844 25554 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 28bfb4a627ad4fe5ae9b3e32ce60831d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:00:11.309441 25566 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "331f5220441645189cb22c6d17b05de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.309682 25566 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 331f5220441645189cb22c6d17b05de0: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:00:11.311669 25552 tablet_replica.cc:439] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "331f5220441645189cb22c6d17b05de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } } }
I20260924 16:00:11.312017 25552 tablet_replica.cc:442] T db0d94e1b7f3498ca8e29407f55b913a P 86662c1918724c8ebdba661787c8ba1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:00:11.328857 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.309949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.16.233.129:36089)
0924 16:00:11.318806 (+ 8857us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.318813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.322592 (+ 3779us) server_negotiation.cc:407] Connection header received
0924 16:00:11.322806 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.322812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.322931 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.323124 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:00:11.323639 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.324842 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.326134 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.326841 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.326969 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.327397 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:00:11.328168 (+ 771us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:00:11.328325 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:11.328374 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 16:00:11.328461 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8699,"thread_start_us":8571,"threads_started":1}
I20260924 16:00:11.345906 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.339418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41455 (local address 127.16.233.129:33893)
0924 16:00:11.340055 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:11.340074 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:11.340243 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:11.340551 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:11.340556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:11.340568 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:00:11.340879 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.340891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.341983 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.341989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.343720 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.343738 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.344833 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.344838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.344960 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.345014 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:00:11.345448 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:00:11.345456 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:00:11.345516 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 16:00:11.345626 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":326,"threads_started":1}
I20260924 16:00:11.345948 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.339458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:33893 (local address 127.16.233.130:41455)
0924 16:00:11.339686 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.339696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.340176 (+ 480us) server_negotiation.cc:407] Connection header received
0924 16:00:11.340368 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.340372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.340438 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.340511 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:00:11.341016 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.341817 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.343955 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.344660 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.344791 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.345215 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:00:11.345345 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:00:11.345442 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:11.345563 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 16:00:11.345656 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:00:11.355229 25477 raft_consensus.cc:1261] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Refusing update from remote peer 28bfb4a627ad4fe5ae9b3e32ce60831d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:11.355608 25330 raft_consensus.cc:1261] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Refusing update from remote peer 28bfb4a627ad4fe5ae9b3e32ce60831d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:11.355865 25578 consensus_queue.cc:1035] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:11.356459 25578 consensus_queue.cc:1035] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [LEADER]: Connected to new peer: Peer: permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:11.399495 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.385087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:50259 (local address 127.16.233.129:36089)
0924 16:00:11.385286 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.385292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.385991 (+ 699us) server_negotiation.cc:407] Connection header received
0924 16:00:11.386203 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.386208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.386326 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.386421 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:00:11.387088 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.388058 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.390138 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.390698 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.390842 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.398718 (+ 7876us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:00:11.398850 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:00:11.398907 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:11.399113 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 16:00:11.399195 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:00:11.400327 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.385152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.16.233.130:50259)
0924 16:00:11.385877 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:11.385897 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:11.386042 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:11.386438 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:11.386444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:11.386459 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:00:11.386811 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.386825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.388193 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.388199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.389988 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:11.390005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.398138 (+ 8133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.398145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:11.398401 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.398496 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:00:11.398987 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:00:11.398997 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:00:11.399763 (+ 766us) client_negotiation.cc:240] Negotiation successful
0924 16:00:11.399930 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":459,"threads_started":1}
I20260924 16:00:11.442512 25027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:11.442809 25027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:11.446563 25027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379: Sending DeleteTablet for 3 replicas of tablet 54b3bd0a0297417b95f3707debeb698e
I20260924 16:00:11.447590 25166 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b3bd0a0297417b95f3707debeb698e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:11 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:55260
I20260924 16:00:11.447820 25451 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b3bd0a0297417b95f3707debeb698e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:11 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:40908
I20260924 16:00:11.448390 25310 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b3bd0a0297417b95f3707debeb698e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:11 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:38496
I20260924 16:00:11.448464 25617 tablet_replica.cc:330] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: stopping tablet replica
I20260924 16:00:11.448668 25617 raft_consensus.cc:2229] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:11.448994 25617 raft_consensus.cc:2258] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:11.450572 25617 ts_tablet_manager.cc:1905] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:11.459687 25617 ts_tablet_manager.cc:1918] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:00:11.459915 25617 log.cc:1196] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/54b3bd0a0297417b95f3707debeb698e
I20260924 16:00:11.460489 25617 ts_tablet_manager.cc:1939] T 54b3bd0a0297417b95f3707debeb698e P 86662c1918724c8ebdba661787c8ba1b: Deleting consensus metadata
I20260924 16:00:11.463104 25014 catalog_manager.cc:4841] TS 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735): tablet 54b3bd0a0297417b95f3707debeb698e (table test-table [id=1d62bc3118c1417bb5e0d398e8a13e87]) successfully deleted
I20260924 16:00:11.473567 25618 tablet_replica.cc:330] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: stopping tablet replica
I20260924 16:00:11.474035 25618 raft_consensus.cc:2229] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:11.474525 25618 raft_consensus.cc:2258] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:11.475567 25616 tablet_replica.cc:330] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: stopping tablet replica
I20260924 16:00:11.475803 25616 raft_consensus.cc:2229] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:00:11.476024 25618 ts_tablet_manager.cc:1905] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:11.476468 25616 raft_consensus.cc:2258] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:11.482553 25616 ts_tablet_manager.cc:1905] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:11.493494 25618 ts_tablet_manager.cc:1918] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:00:11.493777 25618 log.cc:1196] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/54b3bd0a0297417b95f3707debeb698e
I20260924 16:00:11.498870 25616 ts_tablet_manager.cc:1918] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:00:11.499163 25618 ts_tablet_manager.cc:1939] T 54b3bd0a0297417b95f3707debeb698e P 331f5220441645189cb22c6d17b05de0: Deleting consensus metadata
I20260924 16:00:11.500440 25616 log.cc:1196] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/54b3bd0a0297417b95f3707debeb698e
I20260924 16:00:11.500707 25011 catalog_manager.cc:4841] TS 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455): tablet 54b3bd0a0297417b95f3707debeb698e (table test-table [id=1d62bc3118c1417bb5e0d398e8a13e87]) successfully deleted
I20260924 16:00:11.501019 25616 ts_tablet_manager.cc:1939] T 54b3bd0a0297417b95f3707debeb698e P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting consensus metadata
I20260924 16:00:11.502339 25013 catalog_manager.cc:4841] TS 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129:36089): tablet 54b3bd0a0297417b95f3707debeb698e (table test-table [id=1d62bc3118c1417bb5e0d398e8a13e87]) successfully deleted
Sep 24 16:00:11 dist-test-slave-xn4h krb5kdc[24981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:00:11 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:11 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:00:11.569592 25586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:11.553519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.16.233.190:45497)
0924 16:00:11.553720 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.553727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.554242 (+ 515us) server_negotiation.cc:407] Connection header received
0924 16:00:11.554610 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.554618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.554728 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.554849 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:11.556680 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.557906 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.559908 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.560631 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.560785 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.567622 (+ 6837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37162: BlockingRecv error: recv got EOF from 127.0.0.1:37162 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 16:00:11 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM
I20260924 16:00:11.590909 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.571974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.16.233.190:45497)
0924 16:00:11.572150 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.572156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.572178 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:00:11.572418 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.572425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.572512 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.572586 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:11.574410 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.575604 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.577542 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.578249 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.578429 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.583755 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:11.583763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:11.583769 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:11.583825 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:11.587549 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:11.588470 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:11.588476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:11.588480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:11.588642 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:11.589129 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:11.589134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:11.589140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:11.589572 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:11.589834 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:11.590207 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 16:00:11.590538 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:00:11.595398 25027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:00:11.596141 25027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:00:11.604807 25166 tablet_service.cc:1467] Processing CreateTablet for tablet 5585ab3538c846fc97109036d74e7c2c (DEFAULT_TABLE table=test-table [id=289ef24ff2d34e5687d51578c5cee3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:11.605322 25166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5585ab3538c846fc97109036d74e7c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:11.607023 25451 tablet_service.cc:1467] Processing CreateTablet for tablet 5585ab3538c846fc97109036d74e7c2c (DEFAULT_TABLE table=test-table [id=289ef24ff2d34e5687d51578c5cee3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:11.607472 25451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5585ab3538c846fc97109036d74e7c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:11.607522 25310 tablet_service.cc:1467] Processing CreateTablet for tablet 5585ab3538c846fc97109036d74e7c2c (DEFAULT_TABLE table=test-table [id=289ef24ff2d34e5687d51578c5cee3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:11.607919 25310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5585ab3538c846fc97109036d74e7c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:11.612466 25548 tablet_bootstrap.cc:492] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: Bootstrap starting.
I20260924 16:00:11.613961 25546 tablet_bootstrap.cc:492] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: Bootstrap starting.
I20260924 16:00:11.614137 25548 tablet_bootstrap.cc:654] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:11.614195 25547 tablet_bootstrap.cc:492] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: Bootstrap starting.
I20260924 16:00:11.615581 25546 tablet_bootstrap.cc:654] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:11.616034 25547 tablet_bootstrap.cc:654] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:11.625433 25548 tablet_bootstrap.cc:492] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: No bootstrap required, opened a new log
I20260924 16:00:11.625759 25548 ts_tablet_manager.cc:1397] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260924 16:00:11.626761 25548 raft_consensus.cc:348] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.627243 25548 raft_consensus.cc:374] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:11.627421 25548 raft_consensus.cc:729] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bfb4a627ad4fe5ae9b3e32ce60831d, State: Initialized, Role: FOLLOWER
I20260924 16:00:11.627813 25548 consensus_queue.cc:260] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.630927 25548 ts_tablet_manager.cc:1428] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 16:00:11.633337 25547 tablet_bootstrap.cc:492] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: No bootstrap required, opened a new log
I20260924 16:00:11.633515 25547 ts_tablet_manager.cc:1397] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.008s
I20260924 16:00:11.634471 25547 raft_consensus.cc:348] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.634797 25547 raft_consensus.cc:374] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:11.635145 25547 raft_consensus.cc:729] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331f5220441645189cb22c6d17b05de0, State: Initialized, Role: FOLLOWER
I20260924 16:00:11.635437 25546 tablet_bootstrap.cc:492] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: No bootstrap required, opened a new log
I20260924 16:00:11.635685 25546 ts_tablet_manager.cc:1397] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.005s
I20260924 16:00:11.635555 25547 consensus_queue.cc:260] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.636157 25547 ts_tablet_manager.cc:1428] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:11.636670 25546 raft_consensus.cc:348] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.637010 25546 raft_consensus.cc:374] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:11.637162 25546 raft_consensus.cc:729] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86662c1918724c8ebdba661787c8ba1b, State: Initialized, Role: FOLLOWER
I20260924 16:00:11.637491 25546 consensus_queue.cc:260] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.640376 25546 ts_tablet_manager.cc:1428] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:00:11.769411 25554 raft_consensus.cc:482] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:11.769729 25554 raft_consensus.cc:504] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.770470 25554 leader_election.cc:288] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455), 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735)
I20260924 16:00:11.771163 25330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5585ab3538c846fc97109036d74e7c2c" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "331f5220441645189cb22c6d17b05de0" is_pre_election: true
I20260924 16:00:11.771155 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5585ab3538c846fc97109036d74e7c2c" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b" is_pre_election: true
I20260924 16:00:11.771426 25477 raft_consensus.cc:2454] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 0.
I20260924 16:00:11.771426 25330 raft_consensus.cc:2454] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 0.
I20260924 16:00:11.771903 25116 leader_election.cc:302] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bfb4a627ad4fe5ae9b3e32ce60831d, 331f5220441645189cb22c6d17b05de0; no voters:
I20260924 16:00:11.772267 25554 raft_consensus.cc:2781] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:00:11.772396 25554 raft_consensus.cc:482] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:00:11.772518 25554 raft_consensus.cc:3037] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.775764 25554 raft_consensus.cc:504] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.776822 25554 leader_election.cc:288] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 election: Requested vote from peers 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455), 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735)
I20260924 16:00:11.777000 25330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5585ab3538c846fc97109036d74e7c2c" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "331f5220441645189cb22c6d17b05de0"
I20260924 16:00:11.777227 25330 raft_consensus.cc:3037] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.779368 25330 raft_consensus.cc:2454] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 1.
I20260924 16:00:11.779866 25116 leader_election.cc:302] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bfb4a627ad4fe5ae9b3e32ce60831d, 331f5220441645189cb22c6d17b05de0; no voters:
I20260924 16:00:11.780179 25554 raft_consensus.cc:2781] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:11.780272 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5585ab3538c846fc97109036d74e7c2c" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b"
I20260924 16:00:11.780411 25554 raft_consensus.cc:686] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 LEADER]: Becoming Leader. State: Replica: 28bfb4a627ad4fe5ae9b3e32ce60831d, State: Running, Role: LEADER
I20260924 16:00:11.780505 25477 raft_consensus.cc:3037] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.780678 25554 consensus_queue.cc:237] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.784004 25477 raft_consensus.cc:2454] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 1.
I20260924 16:00:11.788811 25028 catalog_manager.cc:5495] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d reported cstate change: term changed from 0 to 1, leader changed from <none> to 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129). New cstate: current_term: 1 leader_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } health_report { overall_health: UNKNOWN } } }
I20260924 16:00:11.822898 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:11.817775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.16.233.129:36089)
0924 16:00:11.817973 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:11.817980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:11.818138 (+ 158us) server_negotiation.cc:407] Connection header received
0924 16:00:11.818406 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:11.818418 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:11.818493 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:11.818595 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:00:11.818954 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.819712 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.821029 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:11.821505 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:11.821657 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:11.822075 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:00:11.822179 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:00:11.822322 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:11.822487 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 16:00:11.822589 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:00:11.826373 25477 raft_consensus.cc:1261] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Refusing update from remote peer 28bfb4a627ad4fe5ae9b3e32ce60831d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:11.826483 25330 raft_consensus.cc:1261] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Refusing update from remote peer 28bfb4a627ad4fe5ae9b3e32ce60831d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:11.827168 25554 consensus_queue.cc:1035] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:11.827517 25611 consensus_queue.cc:1035] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [LEADER]: Connected to new peer: Peer: permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:11.842839 25028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:11.843266 25028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:11.846365 25028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379: Sending DeleteTablet for 3 replicas of tablet 5585ab3538c846fc97109036d74e7c2c
I20260924 16:00:11.847169 25166 tablet_service.cc:1514] Processing DeleteTablet for tablet 5585ab3538c846fc97109036d74e7c2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:11 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:55260
I20260924 16:00:11.847227 25310 tablet_service.cc:1514] Processing DeleteTablet for tablet 5585ab3538c846fc97109036d74e7c2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:11 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:38496
I20260924 16:00:11.847492 25616 tablet_replica.cc:330] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: stopping tablet replica
I20260924 16:00:11.847536 25618 tablet_replica.cc:330] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: stopping tablet replica
I20260924 16:00:11.847679 25616 raft_consensus.cc:2229] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:00:11.847725 25618 raft_consensus.cc:2229] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:11.847934 25618 raft_consensus.cc:2258] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:11.847998 25616 raft_consensus.cc:2258] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:11.847952 25451 tablet_service.cc:1514] Processing DeleteTablet for tablet 5585ab3538c846fc97109036d74e7c2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:11 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:40908
I20260924 16:00:11.848249 25617 tablet_replica.cc:330] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: stopping tablet replica
I20260924 16:00:11.848461 25617 raft_consensus.cc:2229] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:11.848693 25617 raft_consensus.cc:2258] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:11.849853 25617 ts_tablet_manager.cc:1905] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:11.849754 25028 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:00:11.850346 25616 ts_tablet_manager.cc:1905] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:00:11.850446 25028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:00:11.856022 25616 ts_tablet_manager.cc:1918] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:11.856204 25616 log.cc:1196] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/5585ab3538c846fc97109036d74e7c2c
I20260924 16:00:11.856670 25616 ts_tablet_manager.cc:1939] T 5585ab3538c846fc97109036d74e7c2c P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting consensus metadata
I20260924 16:00:11.858490 25013 catalog_manager.cc:4841] TS 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129:36089): tablet 5585ab3538c846fc97109036d74e7c2c (table test-table [id=289ef24ff2d34e5687d51578c5cee3c7]) successfully deleted
I20260924 16:00:11.860939 25617 ts_tablet_manager.cc:1918] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:11.861110 25617 log.cc:1196] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/5585ab3538c846fc97109036d74e7c2c
I20260924 16:00:11.861336 25451 tablet_service.cc:1467] Processing CreateTablet for tablet 78328b54b7494c36b5195ab49f0e8ca4 (DEFAULT_TABLE table=test-table [id=e559f28313f2436ebbb5d26ddf067507]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:11.861575 25617 ts_tablet_manager.cc:1939] T 5585ab3538c846fc97109036d74e7c2c P 86662c1918724c8ebdba661787c8ba1b: Deleting consensus metadata
I20260924 16:00:11.861809 25451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78328b54b7494c36b5195ab49f0e8ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:11.863484 25166 tablet_service.cc:1467] Processing CreateTablet for tablet 78328b54b7494c36b5195ab49f0e8ca4 (DEFAULT_TABLE table=test-table [id=e559f28313f2436ebbb5d26ddf067507]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:11.863898 25166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78328b54b7494c36b5195ab49f0e8ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:11.866909 25014 catalog_manager.cc:4841] TS 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735): tablet 5585ab3538c846fc97109036d74e7c2c (table test-table [id=289ef24ff2d34e5687d51578c5cee3c7]) successfully deleted
I20260924 16:00:11.866998 25310 tablet_service.cc:1467] Processing CreateTablet for tablet 78328b54b7494c36b5195ab49f0e8ca4 (DEFAULT_TABLE table=test-table [id=e559f28313f2436ebbb5d26ddf067507]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:11.867497 25310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78328b54b7494c36b5195ab49f0e8ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:11.867797 25618 ts_tablet_manager.cc:1905] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:11.870680 25548 tablet_bootstrap.cc:492] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: Bootstrap starting.
I20260924 16:00:11.872521 25548 tablet_bootstrap.cc:654] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:11.873543 25547 tablet_bootstrap.cc:492] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: Bootstrap starting.
I20260924 16:00:11.875018 25547 tablet_bootstrap.cc:654] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:11.877682 25547 tablet_bootstrap.cc:492] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: No bootstrap required, opened a new log
I20260924 16:00:11.877872 25547 ts_tablet_manager.cc:1397] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 16:00:11.878252 25618 ts_tablet_manager.cc:1918] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:11.878454 25618 log.cc:1196] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/5585ab3538c846fc97109036d74e7c2c
I20260924 16:00:11.878736 25547 raft_consensus.cc:348] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.878913 25618 ts_tablet_manager.cc:1939] T 5585ab3538c846fc97109036d74e7c2c P 331f5220441645189cb22c6d17b05de0: Deleting consensus metadata
I20260924 16:00:11.878959 25547 raft_consensus.cc:374] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:11.879056 25547 raft_consensus.cc:729] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331f5220441645189cb22c6d17b05de0, State: Initialized, Role: FOLLOWER
I20260924 16:00:11.879354 25547 consensus_queue.cc:260] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.879869 25547 ts_tablet_manager.cc:1428] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 16:00:11.882015 25546 tablet_bootstrap.cc:492] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: Bootstrap starting.
I20260924 16:00:11.883139 25548 tablet_bootstrap.cc:492] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: No bootstrap required, opened a new log
I20260924 16:00:11.883351 25548 ts_tablet_manager.cc:1397] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20260924 16:00:11.883836 25546 tablet_bootstrap.cc:654] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:11.884109 25548 raft_consensus.cc:348] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.884378 25548 raft_consensus.cc:374] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:11.884460 25548 raft_consensus.cc:729] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bfb4a627ad4fe5ae9b3e32ce60831d, State: Initialized, Role: FOLLOWER
I20260924 16:00:11.885838 25548 consensus_queue.cc:260] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.886541 25548 ts_tablet_manager.cc:1428] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 16:00:11.887388 25011 catalog_manager.cc:4841] TS 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455): tablet 5585ab3538c846fc97109036d74e7c2c (table test-table [id=289ef24ff2d34e5687d51578c5cee3c7]) successfully deleted
I20260924 16:00:11.895056 25546 tablet_bootstrap.cc:492] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: No bootstrap required, opened a new log
I20260924 16:00:11.895253 25546 ts_tablet_manager.cc:1397] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260924 16:00:11.896100 25546 raft_consensus.cc:348] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.896471 25546 raft_consensus.cc:374] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:11.896588 25546 raft_consensus.cc:729] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86662c1918724c8ebdba661787c8ba1b, State: Initialized, Role: FOLLOWER
I20260924 16:00:11.896881 25546 consensus_queue.cc:260] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.897466 25546 ts_tablet_manager.cc:1428] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:00:11.949615 25554 raft_consensus.cc:482] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:11.949882 25554 raft_consensus.cc:504] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.950687 25554 leader_election.cc:288] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455), 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735)
I20260924 16:00:11.951334 25476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78328b54b7494c36b5195ab49f0e8ca4" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b" is_pre_election: true
I20260924 16:00:11.951349 25330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78328b54b7494c36b5195ab49f0e8ca4" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "331f5220441645189cb22c6d17b05de0" is_pre_election: true
I20260924 16:00:11.951594 25330 raft_consensus.cc:2454] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 0.
I20260924 16:00:11.951615 25476 raft_consensus.cc:2454] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 0.
I20260924 16:00:11.952132 25116 leader_election.cc:302] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bfb4a627ad4fe5ae9b3e32ce60831d, 331f5220441645189cb22c6d17b05de0; no voters:
I20260924 16:00:11.952476 25554 raft_consensus.cc:2781] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:00:11.952593 25554 raft_consensus.cc:482] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:00:11.952704 25554 raft_consensus.cc:3037] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.956076 25554 raft_consensus.cc:504] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.956890 25554 leader_election.cc:288] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 election: Requested vote from peers 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455), 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735)
I20260924 16:00:11.957397 25476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78328b54b7494c36b5195ab49f0e8ca4" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86662c1918724c8ebdba661787c8ba1b"
I20260924 16:00:11.957489 25330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78328b54b7494c36b5195ab49f0e8ca4" candidate_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "331f5220441645189cb22c6d17b05de0"
I20260924 16:00:11.957660 25476 raft_consensus.cc:3037] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.957736 25330 raft_consensus.cc:3037] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:11.959817 25476 raft_consensus.cc:2454] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 1.
I20260924 16:00:11.960318 25119 leader_election.cc:302] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bfb4a627ad4fe5ae9b3e32ce60831d, 86662c1918724c8ebdba661787c8ba1b; no voters:
I20260924 16:00:11.960664 25554 raft_consensus.cc:2781] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:11.960860 25554 raft_consensus.cc:686] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 LEADER]: Becoming Leader. State: Replica: 28bfb4a627ad4fe5ae9b3e32ce60831d, State: Running, Role: LEADER
I20260924 16:00:11.960760 25330 raft_consensus.cc:2454] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bfb4a627ad4fe5ae9b3e32ce60831d in term 1.
I20260924 16:00:11.961148 25554 consensus_queue.cc:237] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } }
I20260924 16:00:11.964263 25028 catalog_manager.cc:5495] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d reported cstate change: term changed from 0 to 1, leader changed from <none> to 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129). New cstate: current_term: 1 leader_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bfb4a627ad4fe5ae9b3e32ce60831d" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 } health_report { overall_health: UNKNOWN } } }
I20260924 16:00:12.011615 25476 raft_consensus.cc:1261] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Refusing update from remote peer 28bfb4a627ad4fe5ae9b3e32ce60831d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:12.011688 25330 raft_consensus.cc:1261] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Refusing update from remote peer 28bfb4a627ad4fe5ae9b3e32ce60831d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:12.012310 25611 consensus_queue.cc:1035] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86662c1918724c8ebdba661787c8ba1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:12.012791 25611 consensus_queue.cc:1035] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [LEADER]: Connected to new peer: Peer: permanent_uuid: "331f5220441645189cb22c6d17b05de0" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:12.024025 25596 mvcc.cc:205] Tried to move back new op lower bound from 7332927946795384832 to 7332927946595840000. Current Snapshot: MvccSnapshot[applied={T|T < 7332927946795384832}]
I20260924 16:00:12.072254 25028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:12.072458 25028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:12.075376 25028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8d6f190dd644a329f7152290c4e4379: Sending DeleteTablet for 3 replicas of tablet 78328b54b7494c36b5195ab49f0e8ca4
I20260924 16:00:12.076190 25166 tablet_service.cc:1514] Processing DeleteTablet for tablet 78328b54b7494c36b5195ab49f0e8ca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:12 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:55260
I20260924 16:00:12.076182 25310 tablet_service.cc:1514] Processing DeleteTablet for tablet 78328b54b7494c36b5195ab49f0e8ca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:12 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:38496
I20260924 16:00:12.076287 25451 tablet_service.cc:1514] Processing DeleteTablet for tablet 78328b54b7494c36b5195ab49f0e8ca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:12 UTC) from {username='oryx', principal='oryx/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:40908
I20260924 16:00:12.076515 25618 tablet_replica.cc:330] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: stopping tablet replica
I20260924 16:00:12.076517 25616 tablet_replica.cc:330] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: stopping tablet replica
I20260924 16:00:12.076599 25617 tablet_replica.cc:330] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: stopping tablet replica
I20260924 16:00:12.076691 25618 raft_consensus.cc:2229] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:12.076692 25616 raft_consensus.cc:2229] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:00:12.076776 25617 raft_consensus.cc:2229] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:12.076903 25618 raft_consensus.cc:2258] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:12.077006 25617 raft_consensus.cc:2258] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:12.077042 25616 raft_consensus.cc:2258] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:12.078282 25618 ts_tablet_manager.cc:1905] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:12.078418 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 25100
I20260924 16:00:12.078519 25617 ts_tablet_manager.cc:1905] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:12.078600 25616 ts_tablet_manager.cc:1905] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:12.085027 25618 ts_tablet_manager.cc:1918] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:00:12.085027 25616 ts_tablet_manager.cc:1918] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:00:12.085166 25616 log.cc:1196] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/78328b54b7494c36b5195ab49f0e8ca4
I20260924 16:00:12.085157 25618 log.cc:1196] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/78328b54b7494c36b5195ab49f0e8ca4
I20260924 16:00:12.085608 25616 ts_tablet_manager.cc:1939] T 78328b54b7494c36b5195ab49f0e8ca4 P 28bfb4a627ad4fe5ae9b3e32ce60831d: Deleting consensus metadata
I20260924 16:00:12.085608 25618 ts_tablet_manager.cc:1939] T 78328b54b7494c36b5195ab49f0e8ca4 P 331f5220441645189cb22c6d17b05de0: Deleting consensus metadata
I20260924 16:00:12.087049 25617 ts_tablet_manager.cc:1918] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:00:12.087230 25617 log.cc:1196] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/78328b54b7494c36b5195ab49f0e8ca4
I20260924 16:00:12.087450 25013 catalog_manager.cc:4841] TS 28bfb4a627ad4fe5ae9b3e32ce60831d (127.16.233.129:36089): tablet 78328b54b7494c36b5195ab49f0e8ca4 (table test-table [id=e559f28313f2436ebbb5d26ddf067507]) successfully deleted
I20260924 16:00:12.087723 25617 ts_tablet_manager.cc:1939] T 78328b54b7494c36b5195ab49f0e8ca4 P 86662c1918724c8ebdba661787c8ba1b: Deleting consensus metadata
I20260924 16:00:12.087826 25011 catalog_manager.cc:4841] TS 331f5220441645189cb22c6d17b05de0 (127.16.233.130:41455): tablet 78328b54b7494c36b5195ab49f0e8ca4 (table test-table [id=e559f28313f2436ebbb5d26ddf067507]) successfully deleted
I20260924 16:00:12.088725 25014 catalog_manager.cc:4841] TS 86662c1918724c8ebdba661787c8ba1b (127.16.233.131:33735): tablet 78328b54b7494c36b5195ab49f0e8ca4 (table test-table [id=e559f28313f2436ebbb5d26ddf067507]) successfully deleted
Sep 24 16:00:12 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.129@KRBTEST.COM
I20260924 16:00:12.101926 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:12.086875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.16.233.129:36089)
0924 16:00:12.087272 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:12.087278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:12.088323 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 16:00:12.088528 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:12.088535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:12.088608 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:12.088716 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:12.089907 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:12.090691 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:12.091885 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:12.092306 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:12.092442 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:12.096272 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:12.096289 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:12.096306 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:12.096338 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:12.099151 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:12.099836 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:12.099844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:12.099846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:12.099973 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:12.100349 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:12.100354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:12.100358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:12.100733 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:12.101063 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:12.101355 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 16:00:12.101597 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260924 16:00:12.102655 25233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:12.230752 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 25100
I20260924 16:00:12.255004 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 25242
Sep 24 16:00:12 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.130@KRBTEST.COM
I20260924 16:00:12.271518 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:12.255797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.16.233.130:41455)
0924 16:00:12.256283 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:12.256288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:12.256306 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:00:12.256363 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:12.256367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:12.256430 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:12.256534 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:12.257843 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:12.258751 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:12.260004 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:12.260474 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:12.260582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:12.264781 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:12.264797 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:12.264814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:12.264850 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:12.268409 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:12.269089 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:12.269098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:12.269100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:12.269247 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:12.269714 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:12.269722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:12.269726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:12.270170 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:12.270538 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:12.270874 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 16:00:12.271124 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
I20260924 16:00:12.272253 25376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:12.419502 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 25242
I20260924 16:00:12.445196 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 25385
Sep 24 16:00:12 dist-test-slave-xn4h krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.131@KRBTEST.COM
I20260924 16:00:12.460891 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:12.445992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.16.233.131:33735)
0924 16:00:12.446466 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:12.446470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:12.446486 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:00:12.446540 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:12.446544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:12.446610 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:12.446699 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:12.447962 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:12.448649 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:12.450130 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:12.450613 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:12.450713 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:12.455130 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:12.455147 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:12.455164 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:12.455199 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:12.458019 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:12.458816 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:12.458823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:12.458825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:12.458975 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:12.459391 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:12.459394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:12.459396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:12.459764 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:12.459983 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:12.460330 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 16:00:12.460496 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":225,"threads_started":1}
I20260924 16:00:12.461587 25517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:12.558059 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 25385
I20260924 16:00:12.576092 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 24995
I20260924 16:00:12.577265 25068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:12.693810 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 24995
2026-09-24T16:00:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5742 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:00:12 dist-test-slave-xn4h krb5kdc[25657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:00:12 dist-test-slave-xn4h krb5kdc[25657](info): set up 2 sockets
Sep 24 16:00:12 dist-test-slave-xn4h krb5kdc[25657](info): commencing operation
krb5kdc: starting...
W20260924 16:00:13.796236 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:00:13 dist-test-slave-xn4h krb5kdc[25657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:00:14.019135 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36337
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:14.371228 25670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:14.371541 25670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:14.371734 25670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:14.379024 25670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:00:14.379139 25670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:14.379245 25670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:14.379354 25670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:00:14.379439 25670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:00:14.390254 25670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:14.391729 25670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:14.393111 25670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:14.405156 25675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:14.405121 25676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:14.405294 25678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:14.405903 25670 server_base.cc:1034] running on GCE node
I20260924 16:00:14.406814 25670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:14.408320 25670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:14.409561 25670 hybrid_clock.cc:648] HybridClock initialized: now 1790265614409512 us; error 41 us; skew 500 ppm
I20260924 16:00:14.409932 25670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T16:00:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1783 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:00:14 dist-test-slave-xn4h krb5kdc[25684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:00:14 dist-test-slave-xn4h krb5kdc[25684](info): set up 2 sockets
Sep 24 16:00:14 dist-test-slave-xn4h krb5kdc[25684](info): commencing operation
krb5kdc: starting...
W20260924 16:00:15.586290 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:00:15 dist-test-slave-xn4h krb5kdc[25684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:00:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:00:15.817625 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:42479
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:16.157167 25697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:16.157465 25697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:16.157658 25697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:16.165124 25697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:00:16.165251 25697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:16.165305 25697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:16.165365 25697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:00:16.165413 25697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:00:16.175565 25697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:42479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:42479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:16.176877 25697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:16.178242 25697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:16.190086 25702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:16.190050 25705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:16.190018 25703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:16.190762 25697 server_base.cc:1034] running on GCE node
I20260924 16:00:16.191560 25697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:16.193020 25697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:16.194258 25697 hybrid_clock.cc:648] HybridClock initialized: now 1790265616194213 us; error 55 us; skew 500 ppm
I20260924 16:00:16.194674 25697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:16 dist-test-slave-xn4h krb5kdc[25684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265616, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:16.197808 25697 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:00:16.199434 25697 webserver.cc:460] Webserver started at http://127.16.233.190:45695/ using document root <none> and password file <none>
I20260924 16:00:16.200183 25697 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:16.200347 25697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:16.200690 25697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:16.203769 25697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "eb8cd92da5c440b8837bec5a45860046"
format_stamp: "Formatted at 2026-09-24 16:00:16 on dist-test-slave-xn4h"
server_key: "0519153bdcba45b2f4704f57cf85d3a8"
server_key_iv: "301dd7a2b23b8d8552789e97c9303aec"
server_key_version: "encryptionkey@0"
I20260924 16:00:16.204543 25697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "eb8cd92da5c440b8837bec5a45860046"
format_stamp: "Formatted at 2026-09-24 16:00:16 on dist-test-slave-xn4h"
server_key: "0519153bdcba45b2f4704f57cf85d3a8"
server_key_iv: "301dd7a2b23b8d8552789e97c9303aec"
server_key_version: "encryptionkey@0"
I20260924 16:00:16.209820 25697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 16:00:16.213258 25712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:16.214600 25697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:00:16.214782 25697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "eb8cd92da5c440b8837bec5a45860046"
format_stamp: "Formatted at 2026-09-24 16:00:16 on dist-test-slave-xn4h"
server_key: "0519153bdcba45b2f4704f57cf85d3a8"
server_key_iv: "301dd7a2b23b8d8552789e97c9303aec"
server_key_version: "encryptionkey@0"
I20260924 16:00:16.214987 25697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:16.236572 25697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:16.241078 25697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:16.241389 25697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:16.260452 25697 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:42479
I20260924 16:00:16.260474 25764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:42479 every 8 connection(s)
I20260924 16:00:16.261904 25697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:00:16.265812 25765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:16.270159 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 25697
I20260924 16:00:16.270398 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:00:16.271060 17318 external_mini_cluster.cc:1428] Setting key 2f333f11f6906f98de5a657de5aff982
I20260924 16:00:16.276788 25765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: Bootstrap starting.
I20260924 16:00:16.280423 25765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:16.281635 25765 log.cc:824] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:16.285109 25765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: No bootstrap required, opened a new log
Sep 24 16:00:16 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:00:16.292981 25765 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } }
I20260924 16:00:16.293385 25765 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:16.293516 25765 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8cd92da5c440b8837bec5a45860046, State: Initialized, Role: FOLLOWER
I20260924 16:00:16.294229 25765 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } }
I20260924 16:00:16.294554 25765 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:00:16.294711 25765 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:00:16.294898 25765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:16.295264 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:16.273133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.16.233.190:42479)
0924 16:00:16.274001 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:16.274013 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:16.274060 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:00:16.276019 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:16.276044 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:16.276435 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:16.276738 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:16.278656 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:16.279973 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:16.281821 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:16.282937 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:16.283097 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:16.287593 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:16.287603 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:16.287618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:16.287657 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:16.290889 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:16.291599 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:16.291606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:16.291612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:16.291804 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:16.292187 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:16.292194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:16.292201 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:16.292739 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:16.293035 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:16.293501 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 16:00:16.293994 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":395,"threads_started":1}
I20260924 16:00:16.297258 25765 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } }
I20260924 16:00:16.297756 25765 leader_election.cc:302] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb8cd92da5c440b8837bec5a45860046; no voters:
I20260924 16:00:16.298401 25765 leader_election.cc:288] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:00:16.298533 25770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:16.299234 25770 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 1 LEADER]: Becoming Leader. State: Replica: eb8cd92da5c440b8837bec5a45860046, State: Running, Role: LEADER
I20260924 16:00:16.299808 25770 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } }
I20260924 16:00:16.300387 25765 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:00:16.302948 25772 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8cd92da5c440b8837bec5a45860046. Latest consensus state: current_term: 1 leader_uuid: "eb8cd92da5c440b8837bec5a45860046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } } }
I20260924 16:00:16.303469 25771 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb8cd92da5c440b8837bec5a45860046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } } }
I20260924 16:00:16.304870 25772 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:16.305146 25771 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:16.306648 25775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:00:16.311803 25775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:00:16.321058 25775 catalog_manager.cc:1347] Generated new cluster ID: 25f91e7ac9b84d67ae28c4eda7ef2139
I20260924 16:00:16.321193 25775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:00:16.355839 25775 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:00:16.357239 25775 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:00:16.363853 25775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: Generated new TSK 0
I20260924 16:00:16.364720 25775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 16:00:16.421747 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:42479
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:16.754755 25791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:16.755115 25791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:16.755299 25791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:16.762635 25791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:16.762800 25791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:16.762993 25791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:00:16.772850 25791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:42479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:16.774344 25791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:16.775727 25791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:16.789426 25796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:16.789323 25799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:16.789283 25797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:16.790547 25791 server_base.cc:1034] running on GCE node
I20260924 16:00:16.791191 25791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:16.792096 25791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:16.793298 25791 hybrid_clock.cc:648] HybridClock initialized: now 1790265616793248 us; error 40 us; skew 500 ppm
I20260924 16:00:16.793630 25791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:16 dist-test-slave-xn4h krb5kdc[25684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265616, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:16.796854 25791 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:00:16.798560 25791 webserver.cc:460] Webserver started at http://127.16.233.129:36387/ using document root <none> and password file <none>
I20260924 16:00:16.799310 25791 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:16.799465 25791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:16.799789 25791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:16.803344 25791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "c6019fbe730d43b0a2fcf7e03ed1e400"
format_stamp: "Formatted at 2026-09-24 16:00:16 on dist-test-slave-xn4h"
server_key: "6ad25e2aed35a4837f0ffba6efa2897e"
server_key_iv: "5b5613c126ff1d8eb58e2a0458db2903"
server_key_version: "encryptionkey@0"
I20260924 16:00:16.804193 25791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "c6019fbe730d43b0a2fcf7e03ed1e400"
format_stamp: "Formatted at 2026-09-24 16:00:16 on dist-test-slave-xn4h"
server_key: "6ad25e2aed35a4837f0ffba6efa2897e"
server_key_iv: "5b5613c126ff1d8eb58e2a0458db2903"
server_key_version: "encryptionkey@0"
I20260924 16:00:16.810086 25791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:00:16.813673 25806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:16.815184 25791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:16.815378 25791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "c6019fbe730d43b0a2fcf7e03ed1e400"
format_stamp: "Formatted at 2026-09-24 16:00:16 on dist-test-slave-xn4h"
server_key: "6ad25e2aed35a4837f0ffba6efa2897e"
server_key_iv: "5b5613c126ff1d8eb58e2a0458db2903"
server_key_version: "encryptionkey@0"
I20260924 16:00:16.815593 25791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:16.831284 25791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:16.835750 25791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:16.836071 25791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:16.837116 25791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:16.838550 25791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:16.838657 25791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:16.838809 25791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:16.838884 25791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:16.872279 25791 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:36299
I20260924 16:00:16.872334 25919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:36299 every 8 connection(s)
I20260924 16:00:16.873637 25791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:00:16.876617 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 25791
I20260924 16:00:16.876830 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:00:16.877701 17318 external_mini_cluster.cc:1428] Setting key 40f87400c71f8ea95525d18cc588a354
Sep 24 16:00:16 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265616, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:00:16.897192 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:16.876245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:46943 (local address 127.16.233.190:42479)
0924 16:00:16.876806 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:16.876815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:16.877986 (+ 1171us) server_negotiation.cc:407] Connection header received
0924 16:00:16.880206 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:16.880213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:16.880307 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:16.880429 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:16.882050 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:16.883232 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:16.884917 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:16.885657 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:16.885827 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:16.889790 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:16.889798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:16.889805 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:16.889853 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:16.893500 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:16.894396 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:16.894405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:16.894412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:16.894573 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:16.895036 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:16.895044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:16.895050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:16.895489 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:16.895658 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:16.896434 (+ 776us) server_negotiation.cc:299] Negotiation successful
0924 16:00:16.896714 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":252,"threads_started":1}
I20260924 16:00:16.897899 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:16.876909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:46943)
0924 16:00:16.877798 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:16.877844 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:16.879898 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:16.880656 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:16.880665 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:16.881222 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:16.881851 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:16.881871 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:16.883419 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:16.883423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:16.884746 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:16.884761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:16.885915 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:16.885921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:16.886148 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:16.886974 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:16.887005 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:16.889520 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:16.893681 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:16.893690 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:16.893705 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:16.894214 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:16.894692 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:16.894696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:16.894698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:16.894915 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:16.895650 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:16.895657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:16.896009 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 16:00:16.896256 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 16:00:16.896680 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":376,"threads_started":1}
I20260924 16:00:16.899716 25920 heartbeater.cc:347] Connected to a master server at 127.16.233.190:42479
I20260924 16:00:16.900075 25920 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:16.901064 25920 heartbeater.cc:511] Master 127.16.233.190:42479 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 16:00:16.903578 25730 ts_manager.cc:194] Registered new tserver with Master: c6019fbe730d43b0a2fcf7e03ed1e400 (127.16.233.129:36299)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
I20260924 16:00:16.905313 25730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:46943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 16:00:16.934005 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:42479
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:17.261497 25926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:17.261819 25926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:17.261993 25926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:17.269445 25926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:17.269608 25926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:17.269804 25926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:00:17.279788 25926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:42479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:17.281207 25926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:17.282635 25926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:17.296177 25932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:17.296209 25934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:17.296447 25931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:17.297194 25926 server_base.cc:1034] running on GCE node
I20260924 16:00:17.297900 25926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:17.299042 25926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:17.300278 25926 hybrid_clock.cc:648] HybridClock initialized: now 1790265617300213 us; error 64 us; skew 500 ppm
I20260924 16:00:17.300662 25926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:17 dist-test-slave-xn4h krb5kdc[25684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265617, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:17.304278 25926 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:00:17.305868 25926 webserver.cc:460] Webserver started at http://127.16.233.130:38551/ using document root <none> and password file <none>
I20260924 16:00:17.306672 25926 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:17.306841 25926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:17.307171 25926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:17.310475 25926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "4bcffba62473467f85da1b559f8eb807"
format_stamp: "Formatted at 2026-09-24 16:00:17 on dist-test-slave-xn4h"
server_key: "795bc968f4eecbab37e32e275a26affe"
server_key_iv: "873d44d686972990dffd55abec4aeb3f"
server_key_version: "encryptionkey@0"
I20260924 16:00:17.311452 25926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "4bcffba62473467f85da1b559f8eb807"
format_stamp: "Formatted at 2026-09-24 16:00:17 on dist-test-slave-xn4h"
server_key: "795bc968f4eecbab37e32e275a26affe"
server_key_iv: "873d44d686972990dffd55abec4aeb3f"
server_key_version: "encryptionkey@0"
I20260924 16:00:17.317432 25926 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 16:00:17.321462 25942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:17.322970 25926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:17.323164 25926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "4bcffba62473467f85da1b559f8eb807"
format_stamp: "Formatted at 2026-09-24 16:00:17 on dist-test-slave-xn4h"
server_key: "795bc968f4eecbab37e32e275a26affe"
server_key_iv: "873d44d686972990dffd55abec4aeb3f"
server_key_version: "encryptionkey@0"
I20260924 16:00:17.323381 25926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:17.336571 25926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:17.341486 25926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:17.341818 25926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:17.343022 25926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:17.344547 25926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:17.344659 25926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:17.344825 25926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:17.344904 25926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:17.378434 25926 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:38899
I20260924 16:00:17.378453 26055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:38899 every 8 connection(s)
I20260924 16:00:17.379843 25926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:00:17.386519 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 25926
I20260924 16:00:17.386726 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:00:17.387230 17318 external_mini_cluster.cc:1428] Setting key 5371e342dec4e1811dc9040d700c85d4
Sep 24 16:00:17 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265617, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:00:17.403223 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:17.382657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:44751 (local address 127.16.233.190:42479)
0924 16:00:17.382857 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:17.382862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:17.384226 (+ 1364us) server_negotiation.cc:407] Connection header received
0924 16:00:17.386156 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:17.386162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:17.386262 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:17.386439 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:17.388322 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:17.389355 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:17.391993 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:17.392425 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:17.392526 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:17.396833 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:17.396838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:17.396842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:17.396875 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:17.399646 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:17.400475 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:17.400480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:17.400482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:17.400659 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:17.401195 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:17.401200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:17.401220 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:17.401587 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:17.401753 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:17.402768 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0924 16:00:17.402971 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:00:17.404783 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:17.383235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:44751)
0924 16:00:17.384025 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:17.384074 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:17.385823 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:17.386749 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:17.386762 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:17.387428 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:17.388105 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:17.388130 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:17.389531 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:17.389536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:17.391834 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:17.391853 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:17.392670 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:17.392679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:17.393013 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:17.393802 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:17.393839 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:17.396568 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:17.399813 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:17.399824 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:17.399844 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:17.400351 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:17.400795 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:17.400801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:17.400804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:17.401072 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:17.401756 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:17.401768 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:17.402283 (+ 515us) client_negotiation.cc:769] Sending connection context
0924 16:00:17.402665 (+ 382us) client_negotiation.cc:240] Negotiation successful
0924 16:00:17.403284 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":309,"threads_started":1}
I20260924 16:00:17.406371 26056 heartbeater.cc:347] Connected to a master server at 127.16.233.190:42479
I20260924 16:00:17.406723 26056 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:17.407676 26056 heartbeater.cc:511] Master 127.16.233.190:42479 requested a full tablet report, sending...
I20260924 16:00:17.409436 25730 ts_manager.cc:194] Registered new tserver with Master: 4bcffba62473467f85da1b559f8eb807 (127.16.233.130:38899)
I20260924 16:00:17.410265 25730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:44751
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 16:00:17.441933 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:42479
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:17.778424 26062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:17.778775 26062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:17.778954 26062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:17.786372 26062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:17.786549 26062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:17.786748 26062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:00:17.796764 26062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:42479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:17.798182 26062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:17.799604 26062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:17.813336 26067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:17.813390 26070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:17.813410 26068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:17.814617 26062 server_base.cc:1034] running on GCE node
I20260924 16:00:17.815359 26062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:17.816414 26062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:17.817652 26062 hybrid_clock.cc:648] HybridClock initialized: now 1790265617817596 us; error 55 us; skew 500 ppm
I20260924 16:00:17.817988 26062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:17 dist-test-slave-xn4h krb5kdc[25684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265617, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:17.821578 26062 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:00:17.823305 26062 webserver.cc:460] Webserver started at http://127.16.233.131:35471/ using document root <none> and password file <none>
I20260924 16:00:17.824056 26062 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:17.824220 26062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:17.824566 26062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:17.827737 26062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "9fb0a17e72284554abde3aca36e67034"
format_stamp: "Formatted at 2026-09-24 16:00:17 on dist-test-slave-xn4h"
server_key: "1fa64bd585b5b02d3abbc57e6d332192"
server_key_iv: "efbaa3f307399d3fc6e7388e681f45fd"
server_key_version: "encryptionkey@0"
I20260924 16:00:17.828632 26062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "9fb0a17e72284554abde3aca36e67034"
format_stamp: "Formatted at 2026-09-24 16:00:17 on dist-test-slave-xn4h"
server_key: "1fa64bd585b5b02d3abbc57e6d332192"
server_key_iv: "efbaa3f307399d3fc6e7388e681f45fd"
server_key_version: "encryptionkey@0"
I20260924 16:00:17.834120 26062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:00:17.837853 26077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:17.839461 26062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:17.839656 26062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "9fb0a17e72284554abde3aca36e67034"
format_stamp: "Formatted at 2026-09-24 16:00:17 on dist-test-slave-xn4h"
server_key: "1fa64bd585b5b02d3abbc57e6d332192"
server_key_iv: "efbaa3f307399d3fc6e7388e681f45fd"
server_key_version: "encryptionkey@0"
I20260924 16:00:17.839881 26062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:17.861485 26062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:17.865758 26062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:17.866074 26062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:17.867115 26062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:17.868531 26062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:17.868642 26062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:17.868795 26062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:17.868868 26062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:17.903438 26062 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:38007
I20260924 16:00:17.903549 26189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:38007 every 8 connection(s)
I20260924 16:00:17.904778 26062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:00:17.905303 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 26062
I20260924 16:00:17.905498 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:00:17.906093 17318 external_mini_cluster.cc:1428] Setting key 358c61ffaf9f9a071091ef5447190bb8
I20260924 16:00:17.908749 25920 heartbeater.cc:503] Master 127.16.233.190:42479 was elected leader, sending a full tablet report...
Sep 24 16:00:17 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265617, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:00:17.925431 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:17.907717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:55487 (local address 127.16.233.190:42479)
0924 16:00:17.908186 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:17.908190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:17.909379 (+ 1189us) server_negotiation.cc:407] Connection header received
0924 16:00:17.911608 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:17.911611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:17.911676 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:17.911775 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:17.913209 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:17.913977 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:17.915535 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:17.915932 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:17.916030 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:17.919471 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:17.919476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:17.919479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:17.919510 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:17.922722 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:17.923455 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:17.923459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:17.923461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:17.923578 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:17.923956 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:17.923959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:17.923961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:17.924228 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:17.924335 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:17.924980 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 16:00:17.925142 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":259,"threads_started":1}
I20260924 16:00:17.926415 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:17.908388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:55487)
0924 16:00:17.909222 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:17.909281 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:17.911361 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:17.911995 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:17.912003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:17.912495 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:17.913055 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:17.913072 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:17.914096 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:17.914100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:17.915408 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:17.915420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:17.916125 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:17.916129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:17.916325 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:17.916951 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:17.916977 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:17.919264 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:17.922873 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:17.922881 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:17.922898 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:17.923338 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:17.923669 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:17.923672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:17.923674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:17.923859 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:17.924330 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:17.924336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:17.924644 (+ 308us) client_negotiation.cc:769] Sending connection context
0924 16:00:17.924879 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 16:00:17.925264 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":585,"threads_started":1}
I20260924 16:00:17.927855 26190 heartbeater.cc:347] Connected to a master server at 127.16.233.190:42479
I20260924 16:00:17.928166 26190 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:17.928884 26190 heartbeater.cc:511] Master 127.16.233.190:42479 requested a full tablet report, sending...
I20260924 16:00:17.930270 25729 ts_manager.cc:194] Registered new tserver with Master: 9fb0a17e72284554abde3aca36e67034 (127.16.233.131:38007)
I20260924 16:00:17.931226 25729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:55487
I20260924 16:00:17.935256 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab.
Entry for principal oryx/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab.
WARNING: no policy specified for oryx/127.16.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab.
Entry for principal oryx/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab.
WARNING: no policy specified for oryx/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab.
Entry for principal oryx/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab.
I20260924 16:00:18.077206 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 25791
Sep 24 16:00:18 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 16:00:18.093770 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:18.078543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.16.233.129:36299)
0924 16:00:18.079228 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:18.079234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:18.079262 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:00:18.079327 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:18.079334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:18.079542 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:18.079673 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:18.081146 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:18.082166 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:18.083514 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:18.084075 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:18.084174 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:18.088125 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:18.088136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:18.088155 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:18.088209 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:18.090996 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:18.091608 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:18.091615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:18.091617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:18.091739 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:18.092139 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:18.092143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:18.092145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:18.092496 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:18.092812 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:18.093125 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 16:00:18.093423 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":377,"threads_started":1}
I20260924 16:00:18.095135 25915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:18.187897 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 25791
I20260924 16:00:18.204319 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 25926
Sep 24 16:00:18 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:00:18.220590 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:18.205171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.16.233.130:38899)
0924 16:00:18.205723 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:18.205730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:18.205757 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:00:18.205820 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:18.205827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:18.206048 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:18.206185 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:18.207515 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:18.208565 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:18.209773 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:18.210481 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:18.210582 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:18.214748 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:18.214761 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:18.214777 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:18.214815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:18.217744 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:18.218518 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:18.218526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:18.218528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:18.218661 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:18.218988 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:18.218991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:18.218993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:18.219514 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:18.219764 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:18.220040 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:00:18.220291 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":274,"threads_started":1}
I20260924 16:00:18.221863 26051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:18.312382 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 25926
I20260924 16:00:18.328651 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26062
Sep 24 16:00:18 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 16:00:18.346742 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:18.329737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.16.233.131:38007)
0924 16:00:18.330422 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:18.330430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:18.330464 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:00:18.330532 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:18.330540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:18.330759 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:18.330933 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:18.332373 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:18.333434 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:18.334842 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:18.335517 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:18.335698 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:18.340495 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:18.340509 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:18.340525 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:18.340569 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:18.343629 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:18.344383 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:18.344399 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:18.344402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:18.344597 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:18.344982 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:18.344987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:18.344991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:18.345418 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:18.345764 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:18.346056 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 16:00:18.346374 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":316,"threads_started":1}
I20260924 16:00:18.348281 26185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:18.432245 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 26062
I20260924 16:00:18.448534 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 25697
I20260924 16:00:18.449826 25760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:18.554059 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 25697
I20260924 16:00:18.574968 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:42479
--webserver_interface=127.16.233.190
--webserver_port=45695
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:18.900074 26209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:18.900466 26209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:18.900632 26209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:18.908013 26209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:00:18.908172 26209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:18.908286 26209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:18.908385 26209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:00:18.908471 26209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:00:18.918442 26209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:42479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:42479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=45695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:18.920025 26209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:18.921504 26209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:18.933471 26215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:18.933447 26214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:18.933754 26217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:18.934196 26209 server_base.cc:1034] running on GCE node
I20260924 16:00:18.935009 26209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:18.936414 26209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:18.937665 26209 hybrid_clock.cc:648] HybridClock initialized: now 1790265618937594 us; error 56 us; skew 500 ppm
I20260924 16:00:18.938036 26209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:18 dist-test-slave-xn4h krb5kdc[25684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265618, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:18.941424 26209 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:00:18.942981 26209 webserver.cc:460] Webserver started at http://127.16.233.190:45695/ using document root <none> and password file <none>
I20260924 16:00:18.943713 26209 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:18.943872 26209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:18.949237 26209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:00:18.952135 26224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:18.953528 26209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:00:18.953727 26209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "eb8cd92da5c440b8837bec5a45860046"
format_stamp: "Formatted at 2026-09-24 16:00:16 on dist-test-slave-xn4h"
server_key: "0519153bdcba45b2f4704f57cf85d3a8"
server_key_iv: "301dd7a2b23b8d8552789e97c9303aec"
server_key_version: "encryptionkey@0"
I20260924 16:00:18.954782 26209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:18.989017 26209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:18.993546 26209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:18.993839 26209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:19.013573 26209 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:42479
I20260924 16:00:19.013659 26275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:42479 every 8 connection(s)
I20260924 16:00:19.015429 26209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:00:19.016528 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 26209
I20260924 16:00:19.017555 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:36299
--local_ip_for_outbound_sockets=127.16.233.129
--tserver_master_addrs=127.16.233.190:42479
--webserver_port=36387
--webserver_interface=127.16.233.129
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.233.129
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:00:19.025772 26276 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:00:19.029212 26276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: Bootstrap starting.
I20260924 16:00:19.049086 26276 log.cc:824] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:19.057631 26276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:00:19.058243 26276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: Bootstrap complete.
I20260924 16:00:19.067734 26276 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } }
I20260924 16:00:19.068602 26276 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8cd92da5c440b8837bec5a45860046, State: Initialized, Role: FOLLOWER
I20260924 16:00:19.069295 26276 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } }
I20260924 16:00:19.069537 26276 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:00:19.069666 26276 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:00:19.069867 26276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:00:19.073783 26276 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } }
I20260924 16:00:19.074342 26276 leader_election.cc:302] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb8cd92da5c440b8837bec5a45860046; no voters:
I20260924 16:00:19.075089 26276 leader_election.cc:288] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:00:19.075237 26280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:00:19.076372 26280 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [term 2 LEADER]: Becoming Leader. State: Replica: eb8cd92da5c440b8837bec5a45860046, State: Running, Role: LEADER
I20260924 16:00:19.077013 26280 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } }
I20260924 16:00:19.077380 26276 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:00:19.079484 26282 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8cd92da5c440b8837bec5a45860046. Latest consensus state: current_term: 2 leader_uuid: "eb8cd92da5c440b8837bec5a45860046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } } }
I20260924 16:00:19.079864 26282 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:19.080420 26281 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eb8cd92da5c440b8837bec5a45860046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8cd92da5c440b8837bec5a45860046" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 42479 } } }
I20260924 16:00:19.080725 26281 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:19.093020 26287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:00:19.098999 26287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:00:19.100530 26287 catalog_manager.cc:1259] Loaded cluster ID: 25f91e7ac9b84d67ae28c4eda7ef2139
I20260924 16:00:19.100715 26287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:00:19.104827 26287 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:00:19.106496 26287 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb8cd92da5c440b8837bec5a45860046: Loaded TSK: 0
I20260924 16:00:19.108253 26287 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:00:19.371026 26278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:19.371316 26278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:19.371464 26278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:19.379112 26278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:19.379231 26278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:19.379403 26278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:00:19.389596 26278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.129.keytab
--principal=oryx/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:36299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=36387
--tserver_master_addrs=127.16.233.190:42479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:19.391198 26278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:19.392750 26278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:19.408314 26302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:19.408335 26303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:19.408921 26305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:19.409826 26278 server_base.cc:1034] running on GCE node
I20260924 16:00:19.410565 26278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:19.411655 26278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:19.412873 26278 hybrid_clock.cc:648] HybridClock initialized: now 1790265619412835 us; error 39 us; skew 500 ppm
I20260924 16:00:19.413216 26278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265619, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:19.430147 26278 init.cc:377] Logged in from keytab as oryx/127.16.233.129@KRBTEST.COM (short username oryx)
I20260924 16:00:19.432027 26278 webserver.cc:460] Webserver started at http://127.16.233.129:36387/ using document root <none> and password file <none>
I20260924 16:00:19.432777 26278 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:19.432942 26278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:19.438705 26278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:00:19.441594 26312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:19.443491 26278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:19.443667 26278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "c6019fbe730d43b0a2fcf7e03ed1e400"
format_stamp: "Formatted at 2026-09-24 16:00:16 on dist-test-slave-xn4h"
server_key: "6ad25e2aed35a4837f0ffba6efa2897e"
server_key_iv: "5b5613c126ff1d8eb58e2a0458db2903"
server_key_version: "encryptionkey@0"
I20260924 16:00:19.444890 26278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:19.471426 26278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:19.475425 26278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:19.475648 26278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:19.476608 26278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:19.478024 26278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:19.478120 26278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:19.478325 26278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:19.478392 26278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:19.510942 26278 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:36299
I20260924 16:00:19.511031 26425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:36299 every 8 connection(s)
I20260924 16:00:19.512382 26278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:00:19.513278 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 26278
I20260924 16:00:19.513994 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:38899
--local_ip_for_outbound_sockets=127.16.233.130
--tserver_master_addrs=127.16.233.190:42479
--webserver_port=38551
--webserver_interface=127.16.233.130
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.233.130
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:19.533208 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:19.516429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:34163)
0924 16:00:19.517301 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:19.517344 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:19.519149 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:19.520602 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:19.520616 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:19.521244 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:19.522089 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.522119 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.524622 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.524626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.525869 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.525882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.527429 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.527433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.527622 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.528342 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:19.528398 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:19.532242 (+ 3844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":401,"threads_started":1}
W20260924 16:00:19.533897 26426 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:42479 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.233.190:42479: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
W20260924 16:00:19.534052 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:19.516203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:34163 (local address 127.16.233.190:42479)
0924 16:00:19.516974 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:19.516991 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:19.517487 (+ 496us) server_negotiation.cc:407] Connection header received
0924 16:00:19.519746 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:19.519759 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:19.520083 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:19.520468 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:19.522381 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.524424 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.526030 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.527204 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.527410 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.532736 (+ 5326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:34163: BlockingRecv error: recv got EOF from 127.16.233.129:34163 (error 108)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":257,"threads_started":1}
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:19.543081 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:19.534505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:51213)
0924 16:00:19.534771 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:19.534791 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:19.534901 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:19.535356 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:19.535359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:19.535646 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:19.535784 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.535791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.537374 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.537377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.538241 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.538252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.539323 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.539327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.539445 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.540029 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:19.540045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:19.542890 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145}
W20260924 16:00:19.543347 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:19.534808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:51213 (local address 127.16.233.190:42479)
0924 16:00:19.535014 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:19.535023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:19.535046 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:00:19.535114 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:19.535121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:19.535207 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:19.535330 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:19.535923 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.537232 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.538464 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.539190 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.539321 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.543133 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:51213: BlockingRecv error: recv got EOF from 127.16.233.129:51213 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:19.551723 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:19.544002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:44931)
0924 16:00:19.544166 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:19.544191 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:19.544300 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:19.544621 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:19.544625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:19.544925 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:19.545058 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.545065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.546016 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.546021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.546975 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.546986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.547684 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.547691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.547859 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.548426 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:19.548442 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:19.551426 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:00:19.551978 26426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:00:19.552115 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:19.544176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:44931 (local address 127.16.233.190:42479)
0924 16:00:19.544344 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:19.544352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:19.544370 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:00:19.544417 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:19.544422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:19.544497 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:19.544598 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:19.545199 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.545878 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.547120 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.547534 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.547667 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.551615 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:44931: BlockingRecv error: recv got EOF from 127.16.233.129:44931 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 16:00:19.847160 26430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:19.847486 26430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:19.847661 26430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:19.855131 26430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:19.855297 26430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:19.855485 26430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:00:19.865490 26430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.130.keytab
--principal=oryx/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:38899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=38551
--tserver_master_addrs=127.16.233.190:42479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:19.866865 26430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:19.868292 26430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:19.881345 26436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:19.881465 26435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:19.881418 26438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:19.882386 26430 server_base.cc:1034] running on GCE node
I20260924 16:00:19.882930 26430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:19.883837 26430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:19.885051 26430 hybrid_clock.cc:648] HybridClock initialized: now 1790265619884976 us; error 63 us; skew 500 ppm
I20260924 16:00:19.885411 26430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265619, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:19.888594 26430 init.cc:377] Logged in from keytab as oryx/127.16.233.130@KRBTEST.COM (short username oryx)
I20260924 16:00:19.890120 26430 webserver.cc:460] Webserver started at http://127.16.233.130:38551/ using document root <none> and password file <none>
I20260924 16:00:19.890838 26430 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:19.890954 26430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:19.896304 26430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 16:00:19.899073 26445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:19.900435 26430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:19.900624 26430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "4bcffba62473467f85da1b559f8eb807"
format_stamp: "Formatted at 2026-09-24 16:00:17 on dist-test-slave-xn4h"
server_key: "795bc968f4eecbab37e32e275a26affe"
server_key_iv: "873d44d686972990dffd55abec4aeb3f"
server_key_version: "encryptionkey@0"
I20260924 16:00:19.901574 26430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:19.926436 26430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:19.931021 26430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:19.931324 26430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:19.932400 26430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:19.933763 26430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:19.933887 26430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:19.934031 26430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:19.934114 26430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:19.967815 26430 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:38899
I20260924 16:00:19.967904 26561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:38899 every 8 connection(s)
I20260924 16:00:19.969272 26430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:00:19.979524 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 26430
I20260924 16:00:19.980165 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:38007
--local_ip_for_outbound_sockets=127.16.233.131
--tserver_master_addrs=127.16.233.190:42479
--webserver_port=35471
--webserver_interface=127.16.233.131
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.233.131
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:00:19.986207 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:19.971878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:57859 (local address 127.16.233.190:42479)
0924 16:00:19.972061 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:19.972066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:19.973175 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 16:00:19.975114 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:19.975121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:19.975223 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:19.975336 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:19.976861 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.977690 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.979398 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.979873 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.980007 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.985987 (+ 5980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:57859: BlockingRecv error: recv got EOF from 127.16.233.130:57859 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:00:19.987112 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:19.972245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:57859)
0924 16:00:19.972978 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:19.973022 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:19.974836 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:19.975583 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:19.975591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:19.976126 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:19.976686 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.976703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.977864 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.977868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.979256 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.979269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.980132 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.980141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.980463 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.981154 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:19.981179 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:19.985821 (+ 4642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":271,"threads_started":1}
W20260924 16:00:19.987922 26562 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:42479 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.233.190:42479: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:19 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:19.998203 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:19.988504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:42559)
0924 16:00:19.988722 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:19.988743 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:19.988852 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:19.989172 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:19.989179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:19.989629 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:19.989840 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.989854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.990893 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.990902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.992609 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:19.992628 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.993291 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.993299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:19.993523 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.994133 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:19.994160 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:19.997819 (+ 3659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 16:00:19.998349 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:19.988655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:42559 (local address 127.16.233.190:42479)
0924 16:00:19.988818 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:19.988822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:19.988838 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:00:19.988968 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:19.988972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:19.989031 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:19.989110 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:19.989975 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.990754 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.992755 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:19.993135 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:19.993229 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:19.998063 (+ 4834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:42559: BlockingRecv error: recv got EOF from 127.16.233.130:42559 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:20.007062 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:19.999081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:43713)
0924 16:00:19.999361 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:19.999379 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:19.999481 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:19.999798 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:19.999803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:20.000115 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:20.000260 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.000267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.001222 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.001228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.002108 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.002119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.002862 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.002868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.003034 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.003617 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:20.003634 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:20.006775 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162}
W20260924 16:00:20.007112 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:19.999252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:43713 (local address 127.16.233.190:42479)
0924 16:00:19.999417 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:19.999422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:19.999439 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:00:19.999617 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:19.999621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:19.999676 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:19.999778 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:20.000396 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.001089 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.002252 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.002704 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.002840 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.006959 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:43713: BlockingRecv error: recv got EOF from 127.16.233.130:43713 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 16:00:20.007539 26562 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:00:20.318966 26565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:20.319289 26565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:20.319480 26565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:20.327250 26565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:20.327382 26565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:20.327585 26565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:00:20.337889 26565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/krb5kdc/oryx_127.16.233.131.keytab
--principal=oryx/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:38007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=35471
--tserver_master_addrs=127.16.233.190:42479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:20.339437 26565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:20.340849 26565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:20.354951 26571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:20.355006 26574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:20.354878 26572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:20.356108 26565 server_base.cc:1034] running on GCE node
I20260924 16:00:20.356803 26565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:20.357870 26565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:20.359117 26565 hybrid_clock.cc:648] HybridClock initialized: now 1790265620359065 us; error 65 us; skew 500 ppm
I20260924 16:00:20.359499 26565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265620, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:20.362874 26565 init.cc:377] Logged in from keytab as oryx/127.16.233.131@KRBTEST.COM (short username oryx)
I20260924 16:00:20.364631 26565 webserver.cc:460] Webserver started at http://127.16.233.131:35471/ using document root <none> and password file <none>
I20260924 16:00:20.365424 26565 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:20.365595 26565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:20.371608 26565 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:00:20.374697 26581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:20.376154 26565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:00:20.376384 26565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "9fb0a17e72284554abde3aca36e67034"
format_stamp: "Formatted at 2026-09-24 16:00:17 on dist-test-slave-xn4h"
server_key: "1fa64bd585b5b02d3abbc57e6d332192"
server_key_iv: "efbaa3f307399d3fc6e7388e681f45fd"
server_key_version: "encryptionkey@0"
I20260924 16:00:20.377421 26565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:20.402865 26565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:20.407366 26565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:20.407641 26565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:20.408660 26565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:20.409993 26565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:20.410117 26565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:20.410249 26565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:20.410356 26565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:20.444380 26565 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:38007
I20260924 16:00:20.444471 26694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:38007 every 8 connection(s)
I20260924 16:00:20.445845 26565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:00:20.446219 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 26565
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:00:20.462610 26698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:20.448783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:56259 (local address 127.16.233.190:42479)
0924 16:00:20.449239 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:20.449243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:20.450474 (+ 1231us) server_negotiation.cc:407] Connection header received
0924 16:00:20.452648 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:20.452653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:20.452730 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:20.452863 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:20.454511 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.455253 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.456743 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.457201 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.457342 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.462227 (+ 4885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:56259: BlockingRecv error: recv got EOF from 127.16.233.131:56259 (error 108)
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":270,"threads_started":1}
I20260924 16:00:20.462760 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:20.447270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.16.233.190:42479)
0924 16:00:20.447474 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:20.447481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:20.448044 (+ 563us) server_negotiation.cc:407] Connection header received
0924 16:00:20.448226 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:20.448230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:20.448295 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:20.448389 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:20.450496 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.451576 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.452822 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.453278 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.453415 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.456090 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:20.456098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:20.456111 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:20.456146 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:20.459526 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:20.460262 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:20.460270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:20.460282 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:20.460440 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:20.460843 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:20.460847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:20.460849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:20.461281 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:20.461608 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:20.462070 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 16:00:20.462443 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:00:20.463543 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:20.449457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:56259)
0924 16:00:20.450284 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:20.450356 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:20.452412 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:20.453098 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:20.453108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:20.453588 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:20.454290 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.454354 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.455396 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.455402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.456582 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.456594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.457419 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.457429 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.457702 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.458264 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:20.458284 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:20.461996 (+ 3712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":320,"threads_started":1}
W20260924 16:00:20.464183 26695 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:42479 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.233.190:42479: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:20.472837 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:20.464741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:54735)
0924 16:00:20.464927 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:20.464945 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:20.465045 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:20.465532 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:20.465537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:20.465847 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:20.466013 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.466024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.467324 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.467327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.468218 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.468229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.468951 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.468955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.469081 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.469698 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:20.469715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:20.472548 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 16:00:20.473013 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:20.464874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:54735 (local address 127.16.233.190:42479)
0924 16:00:20.465252 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:20.465256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:20.465269 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:00:20.465306 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:20.465309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:20.465390 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:20.465498 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:20.466155 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.467166 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.468370 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.468822 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.468928 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.472768 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:54735: BlockingRecv error: recv got EOF from 127.16.233.131:54735 (error 108)
Metrics: {"server-negotiator.queue_time_us":261}
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:20.481220 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:20.473585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:36169)
0924 16:00:20.473752 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:20.473767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:20.473850 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:20.474164 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:20.474167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:20.474525 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:20.474667 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.474675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.475574 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.475577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.476479 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.476490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.477155 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.477158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.477279 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.477971 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:20.477992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:20.480987 (+ 2995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 16:00:20.481446 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:20.473717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:36169 (local address 127.16.233.190:42479)
0924 16:00:20.473944 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:20.473948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:20.473962 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:00:20.474004 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:20.474007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:20.474057 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:20.474131 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:20.474783 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.475461 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.476601 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.477023 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.477155 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.481178 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:36169: BlockingRecv error: recv got EOF from 127.16.233.131:36169 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
W20260924 16:00:20.481585 26695 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:20 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:20.561278 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:20.552648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:41577)
0924 16:00:20.553250 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:20.553273 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:20.553416 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:20.553725 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:20.553728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:20.554199 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:20.554411 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.554420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.555354 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.555358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.556313 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:20.556325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.557032 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.557036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:20.557188 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.557844 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:20.557881 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:20.560941 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":285,"threads_started":1}
W20260924 16:00:20.561388 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:20.552733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:41577 (local address 127.16.233.190:42479)
0924 16:00:20.552932 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:20.552941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:20.553371 (+ 430us) server_negotiation.cc:407] Connection header received
0924 16:00:20.553529 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:20.553532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:20.553594 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:20.553671 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:20.554536 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.555231 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.556441 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:20.556856 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:20.556957 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:20.561157 (+ 4200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:41577: BlockingRecv error: recv got EOF from 127.16.233.129:41577 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 16:00:21 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:21 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:21.017060 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:21.008246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:52865)
0924 16:00:21.008838 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:21.008859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:21.008998 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:21.009391 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:21.009395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:21.009963 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:21.010158 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:21.010167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.011200 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.011208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:21.012153 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:21.012165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.012934 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.012939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:21.013112 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:21.013744 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:21.013762 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:21.016728 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":288,"threads_started":1}
W20260924 16:00:21.017234 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:21.008291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:52865 (local address 127.16.233.190:42479)
0924 16:00:21.008522 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:21.008530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:21.008952 (+ 422us) server_negotiation.cc:407] Connection header received
0924 16:00:21.009177 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:21.009181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:21.009246 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:21.009355 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:21.010333 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.011054 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.012298 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.012753 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.012901 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:21.016956 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:52865: BlockingRecv error: recv got EOF from 127.16.233.130:52865 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 16:00:21 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:21 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:21.490911 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:21.482164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:42033)
0924 16:00:21.482706 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:21.482722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:21.482832 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:21.483129 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:21.483133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:21.483569 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:21.483850 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:21.483862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.484888 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.484894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:21.486297 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:21.486333 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.487082 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.487090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:21.487273 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:21.487824 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:21.487842 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:21.490581 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":292,"threads_started":1}
W20260924 16:00:21.491055 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:21.482233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:42033 (local address 127.16.233.190:42479)
0924 16:00:21.482442 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:21.482447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:21.482790 (+ 343us) server_negotiation.cc:407] Connection header received
0924 16:00:21.482932 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:21.482935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:21.482993 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:21.483072 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:21.483969 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.484702 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.486444 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.486888 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.487042 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:21.490782 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:42033: BlockingRecv error: recv got EOF from 127.16.233.131:42033 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 24 16:00:21 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:21 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:21.570574 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:21.562225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:51873)
0924 16:00:21.562799 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:21.562815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:21.562927 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:21.563270 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:21.563274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:21.563716 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:21.563887 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:21.563896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.564897 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.564901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:21.565790 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:21.565801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.566539 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.566544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:21.566666 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:21.567210 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:21.567228 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:21.570277 (+ 3049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":332,"threads_started":1}
W20260924 16:00:21.570744 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:21.562270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:51873 (local address 127.16.233.190:42479)
0924 16:00:21.562455 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:21.562460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:21.562890 (+ 430us) server_negotiation.cc:407] Connection header received
0924 16:00:21.563027 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:21.563030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:21.563115 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:21.563194 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:21.564021 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.564774 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.565911 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:21.566403 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:21.566496 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:21.570511 (+ 4015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:51873: BlockingRecv error: recv got EOF from 127.16.233.129:51873 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 16:00:22 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:22 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:22.027134 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:22.018010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:53999)
0924 16:00:22.018564 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:22.018579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:22.018742 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:22.019106 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:22.019111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:22.019542 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:22.019717 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:22.019726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.020725 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.020737 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:22.021647 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:22.021664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.022639 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.022645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:22.022819 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:22.023511 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:22.023539 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:22.026910 (+ 3371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":334,"threads_started":1}
W20260924 16:00:22.027315 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:22.018111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:53999 (local address 127.16.233.190:42479)
0924 16:00:22.018354 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:22.018362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:22.018667 (+ 305us) server_negotiation.cc:407] Connection header received
0924 16:00:22.018875 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:22.018879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:22.018958 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:22.019065 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:22.019863 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.020573 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.021801 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.022487 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.022609 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:22.027108 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:53999: BlockingRecv error: recv got EOF from 127.16.233.130:53999 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 16:00:22 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:22 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:22.500420 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:22.491912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:35897)
0924 16:00:22.492420 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:22.492436 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:22.492559 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:22.492935 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:22.492939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:22.493327 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:22.493495 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:22.493503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.494542 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.494545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:22.495441 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:22.495452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.496370 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.496376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:22.496525 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:22.497282 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:22.497301 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:22.500195 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":292,"threads_started":1}
W20260924 16:00:22.500590 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:22.492028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:35897 (local address 127.16.233.190:42479)
0924 16:00:22.492279 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:22.492285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:22.492509 (+ 224us) server_negotiation.cc:407] Connection header received
0924 16:00:22.492677 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:22.492681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:22.492757 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:22.492874 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:22.493617 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.494419 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.495578 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.496213 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.496309 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:22.500372 (+ 4063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:35897: BlockingRecv error: recv got EOF from 127.16.233.131:35897 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 24 16:00:22 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:22 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:22.580534 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:22.571665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:47947)
0924 16:00:22.572184 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:22.572199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:22.572359 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:22.572722 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:22.572727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:22.573094 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:22.573260 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:22.573269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.574362 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.574368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:22.575294 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:22.575305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.576031 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.576037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:22.576219 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:22.576908 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:22.576924 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:22.580170 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":286,"threads_started":1}
W20260924 16:00:22.580662 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:22.571739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:47947 (local address 127.16.233.190:42479)
0924 16:00:22.571989 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:22.571998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:22.572288 (+ 290us) server_negotiation.cc:407] Connection header received
0924 16:00:22.572489 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:22.572500 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:22.572560 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:22.572663 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:22.573409 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.574186 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.575438 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:22.575884 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:22.576020 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:22.580385 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:47947: BlockingRecv error: recv got EOF from 127.16.233.129:47947 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 24 16:00:23 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:23 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:23.037009 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:23.028308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:47653)
0924 16:00:23.028956 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:23.028972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:23.029132 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:23.029540 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:23.029545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:23.029946 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:23.030111 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:23.030120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.031227 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.031234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:23.032140 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:23.032152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.032876 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.032881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:23.033075 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:23.033742 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:23.033760 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:23.036642 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":312,"threads_started":1}
W20260924 16:00:23.037097 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:23.028721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:47653 (local address 127.16.233.190:42479)
0924 16:00:23.028912 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:23.028918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:23.029068 (+ 150us) server_negotiation.cc:407] Connection header received
0924 16:00:23.029273 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:23.029279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:23.029393 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:23.029509 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:23.030270 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.031090 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.032297 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.032718 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.032865 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:23.036851 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:47653: BlockingRecv error: recv got EOF from 127.16.233.130:47653 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 16:00:23 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:23 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:23.510509 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:23.501602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:60253)
0924 16:00:23.502132 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:23.502149 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:23.502267 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:23.502667 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:23.502670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:23.503013 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:23.503181 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:23.503189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.504229 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.504232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:23.505341 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:23.505356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.506389 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.506396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:23.506588 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:23.507294 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:23.507313 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:23.510166 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":294,"threads_started":1}
W20260924 16:00:23.510867 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:23.501729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:60253 (local address 127.16.233.190:42479)
0924 16:00:23.501923 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:23.501928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:23.502245 (+ 317us) server_negotiation.cc:407] Connection header received
0924 16:00:23.502460 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:23.502465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:23.502534 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:23.502673 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:23.503337 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.504084 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.505512 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.506145 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.506287 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:23.510470 (+ 4183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:60253: BlockingRecv error: recv got EOF from 127.16.233.131:60253 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 16:00:23 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:23 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:23.589857 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:23.581682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:40743)
0924 16:00:23.582260 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:23.582277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:23.582505 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:23.582824 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:23.582827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:23.583249 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:23.583413 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:23.583422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.584424 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.584427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:23.585374 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:23.585385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.586068 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.586072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:23.586192 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:23.586856 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:23.586874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:23.589622 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":332,"threads_started":1}
W20260924 16:00:23.590173 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:23.581750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:40743 (local address 127.16.233.190:42479)
0924 16:00:23.581931 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:23.581936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:23.582443 (+ 507us) server_negotiation.cc:407] Connection header received
0924 16:00:23.582628 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:23.582631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:23.582697 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:23.582784 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:23.583555 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.584311 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.585512 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:23.585936 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:23.586035 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:23.589855 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:40743: BlockingRecv error: recv got EOF from 127.16.233.129:40743 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 16:00:24 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:24 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:24.045899 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:24.038089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:44819)
0924 16:00:24.038595 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:24.038616 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:24.038731 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:24.039104 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:24.039108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:24.039473 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:24.039618 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:24.039626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.040594 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.040598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:24.041535 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:24.041546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.042174 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.042178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:24.042369 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:24.042930 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:24.042946 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:24.045668 (+ 2722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":309,"threads_started":1}
W20260924 16:00:24.046118 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:24.038159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:44819 (local address 127.16.233.190:42479)
0924 16:00:24.038427 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:24.038433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:24.038695 (+ 262us) server_negotiation.cc:407] Connection header received
0924 16:00:24.038844 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:24.038848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:24.038969 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:24.039058 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:24.039740 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.040467 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.041660 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.042052 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.042154 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:24.045871 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:44819: BlockingRecv error: recv got EOF from 127.16.233.130:44819 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 16:00:24 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:24 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:24.531441 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:24.511585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:38081)
0924 16:00:24.512142 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:24.512160 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:24.512340 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:24.512757 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:24.512762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:24.513276 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:24.513513 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:24.513527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.515082 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.515088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:24.516163 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:24.516175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.516988 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.516995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:24.517198 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:24.528119 (+ 10921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:24.528145 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:24.531192 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":299,"threads_started":1}
W20260924 16:00:24.531749 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:24.511718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:38081 (local address 127.16.233.190:42479)
0924 16:00:24.511997 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:24.512004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:24.512266 (+ 262us) server_negotiation.cc:407] Connection header received
0924 16:00:24.512502 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:24.512507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:24.512590 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:24.512715 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:24.513769 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.514919 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.516324 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.516821 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.516974 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:24.531434 (+ 14460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:38081: BlockingRecv error: recv got EOF from 127.16.233.131:38081 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 24 16:00:24 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:24 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:24.598924 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:24.590868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:44953)
0924 16:00:24.591353 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:24.591369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:24.591479 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:24.591824 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:24.591828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:24.592222 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:24.592402 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:24.592411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.593443 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.593447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:24.594450 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:24.594463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.595121 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.595125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:24.595239 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:24.595822 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:24.595839 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:24.598665 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
W20260924 16:00:24.599236 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:24.590912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:44953 (local address 127.16.233.190:42479)
0924 16:00:24.591174 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:24.591181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:24.591446 (+ 265us) server_negotiation.cc:407] Connection header received
0924 16:00:24.591613 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:24.591617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:24.591684 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:24.591766 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:24.592534 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.593316 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.594589 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:24.594997 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:24.595094 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:24.598990 (+ 3896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:44953: BlockingRecv error: recv got EOF from 127.16.233.129:44953 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 16:00:25 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:25 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:25.055476 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:25.046889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:37953)
0924 16:00:25.047500 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:25.047516 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:25.047625 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:25.047933 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:25.047936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:25.048342 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:25.048509 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:25.048517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.049636 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.049642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:25.050725 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:25.050738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.051370 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.051375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:25.051494 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:25.052059 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:25.052077 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:25.055094 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":333,"threads_started":1}
W20260924 16:00:25.055555 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:25.047032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:37953 (local address 127.16.233.190:42479)
0924 16:00:25.047206 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:25.047210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:25.047589 (+ 379us) server_negotiation.cc:407] Connection header received
0924 16:00:25.047727 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:25.047731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:25.047804 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:25.047886 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:25.048644 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.049493 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.050845 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.051237 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.051339 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:25.055314 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:37953: BlockingRecv error: recv got EOF from 127.16.233.130:37953 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 16:00:25 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:25 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:25.541453 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:25.532398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:44085)
0924 16:00:25.533008 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:25.533025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:25.533190 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:25.533573 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:25.533578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:25.533999 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:25.534158 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:25.534168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.535652 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.535658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:25.536748 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:25.536761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.537484 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.537488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:25.537609 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:25.538296 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:25.538336 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:25.541197 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":343,"threads_started":1}
W20260924 16:00:25.541677 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:25.532534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:44085 (local address 127.16.233.190:42479)
0924 16:00:25.532796 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:25.532804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:25.533154 (+ 350us) server_negotiation.cc:407] Connection header received
0924 16:00:25.533344 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:25.533349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:25.533429 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:25.533549 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:25.534347 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.535412 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.536889 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.537343 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.537443 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:25.541391 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:44085: BlockingRecv error: recv got EOF from 127.16.233.131:44085 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 24 16:00:25 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:25 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:25.608458 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:25.599945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:56633)
0924 16:00:25.600521 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:25.600537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:25.600715 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:25.601074 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:25.601078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:25.601494 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:25.601648 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:25.601656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.602739 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.602744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:25.603620 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:25.603640 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.604371 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.604377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:25.604573 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:25.605168 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:25.605186 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:25.608182 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":338,"threads_started":1}
W20260924 16:00:25.608772 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:25.600034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:56633 (local address 127.16.233.190:42479)
0924 16:00:25.600292 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:25.600299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:25.600629 (+ 330us) server_negotiation.cc:407] Connection header received
0924 16:00:25.600844 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:25.600855 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:25.600936 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:25.601039 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:25.601796 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.602606 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.603771 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:25.604216 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:25.604353 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:25.608487 (+ 4134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:56633: BlockingRecv error: recv got EOF from 127.16.233.129:56633 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 16:00:26 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:26 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:26.064590 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:26.056744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:58799)
0924 16:00:26.057295 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:26.057318 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:26.057433 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:26.057818 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:26.057822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:26.058240 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:26.058437 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:26.058446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.059485 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.059489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:26.060399 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:26.060411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.061039 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.061043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:26.061167 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:26.061691 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:26.061708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:26.064372 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":310,"threads_started":1}
W20260924 16:00:26.064782 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:26.056812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:58799 (local address 127.16.233.190:42479)
0924 16:00:26.057001 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:26.057009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:26.057400 (+ 391us) server_negotiation.cc:407] Connection header received
0924 16:00:26.057568 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:26.057576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:26.057679 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:26.057768 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:26.058558 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.059353 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.060517 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.060906 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.060998 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:26.064563 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:58799: BlockingRecv error: recv got EOF from 127.16.233.130:58799 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 16:00:26 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:26 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:26.550699 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:26.542514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:45591)
0924 16:00:26.543050 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:26.543067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:26.543181 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:26.543479 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:26.543482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:26.543856 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:26.544019 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:26.544028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.544998 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.545001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:26.545878 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:26.545889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.546768 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.546772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:26.546889 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:26.547541 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:26.547559 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:26.550375 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":302,"threads_started":1}
W20260924 16:00:26.550871 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:26.542545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:45591 (local address 127.16.233.190:42479)
0924 16:00:26.542728 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:26.542733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:26.543144 (+ 411us) server_negotiation.cc:407] Connection header received
0924 16:00:26.543289 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:26.543292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:26.543354 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:26.543434 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:26.544135 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.544877 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.545995 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.546622 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.546714 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:26.550628 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:45591: BlockingRecv error: recv got EOF from 127.16.233.131:45591 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 16:00:26 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:26 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:26.625026 26718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:26.609468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:46627)
0924 16:00:26.610043 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:26.610061 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:26.610194 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:26.610623 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:26.610628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:26.611180 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:26.611355 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:26.611366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.612679 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.612683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:26.613638 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:26.613651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.614349 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.614359 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:26.614498 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:26.621152 (+ 6654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:26.621175 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:26.624698 (+ 3523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":279,"threads_started":1}
W20260924 16:00:26.625156 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:26.609587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:46627 (local address 127.16.233.190:42479)
0924 16:00:26.609870 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:26.609877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:26.610169 (+ 292us) server_negotiation.cc:407] Connection header received
0924 16:00:26.610347 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:26.610352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:26.610440 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:26.610564 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:26.611512 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.612540 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.613764 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:26.614176 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:26.614348 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:26.624936 (+ 10588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:46627: BlockingRecv error: recv got EOF from 127.16.233.129:46627 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 24 16:00:27 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:27 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:27.074124 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:27.065569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:47503)
0924 16:00:27.066098 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:27.066114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:27.066230 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:27.066575 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:27.066579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:27.066977 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:27.067143 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:27.067151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.068102 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.068105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:27.069012 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:27.069023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.069717 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.069721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:27.069844 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:27.070551 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:27.070577 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:27.073849 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":246,"threads_started":1}
W20260924 16:00:27.074560 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:27.065697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:47503 (local address 127.16.233.190:42479)
0924 16:00:27.065954 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:27.065961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:27.066198 (+ 237us) server_negotiation.cc:407] Connection header received
0924 16:00:27.066380 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:27.066384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:27.066448 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:27.066530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:27.067269 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.067985 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.069150 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.069569 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.069683 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:27.074250 (+ 4567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:47503: BlockingRecv error: recv got EOF from 127.16.233.130:47503 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 16:00:27 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:27 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:27.561040 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:27.551876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:48483)
0924 16:00:27.552559 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:27.552578 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:27.552704 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:27.553037 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:27.553041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:27.553614 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:27.553896 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:27.553913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.555316 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.555320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:27.556235 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:27.556247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.556940 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.556945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:27.557061 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:27.557734 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:27.557752 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:27.560676 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":320,"threads_started":1}
W20260924 16:00:27.561139 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:27.551967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:48483 (local address 127.16.233.190:42479)
0924 16:00:27.552161 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:27.552166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:27.552710 (+ 544us) server_negotiation.cc:407] Connection header received
0924 16:00:27.552830 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:27.552834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:27.552899 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:27.553035 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:27.554089 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.555185 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.556360 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.556776 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.556884 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:27.560907 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:48483: BlockingRecv error: recv got EOF from 127.16.233.131:48483 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 16:00:27 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:27 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:27.634442 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:27.626037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:39025)
0924 16:00:27.626889 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:27.626906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:27.627056 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:27.627446 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:27.627448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:27.627846 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:27.627998 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:27.628006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.629034 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.629038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:27.629927 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:27.629937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.630697 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.630701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:27.630837 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:27.631410 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:27.631427 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:27.634118 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":585,"threads_started":1}
W20260924 16:00:27.634720 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:27.626193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:39025 (local address 127.16.233.190:42479)
0924 16:00:27.626459 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:27.626469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:27.626973 (+ 504us) server_negotiation.cc:407] Connection header received
0924 16:00:27.627244 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:27.627248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:27.627313 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:27.627399 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:27.628140 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.628912 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.630056 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:27.630548 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:27.630643 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:27.634434 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:39025: BlockingRecv error: recv got EOF from 127.16.233.129:39025 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 16:00:28 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:28 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:28.084190 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:28.075118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:42231)
0924 16:00:28.075637 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:28.075653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:28.075809 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:28.076176 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:28.076181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:28.076586 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:28.076734 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:28.076743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.078042 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.078048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:28.079096 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:28.079110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.079856 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.079862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:28.080022 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:28.080670 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:28.080688 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:28.083970 (+ 3282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":292,"threads_started":1}
W20260924 16:00:28.084363 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:28.075196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:42231 (local address 127.16.233.190:42479)
0924 16:00:28.075401 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:28.075409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:28.075738 (+ 329us) server_negotiation.cc:407] Connection header received
0924 16:00:28.075933 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:28.075945 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:28.076007 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:28.076107 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:28.076879 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.077885 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.079238 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.079670 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.079799 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:28.084157 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:42231: BlockingRecv error: recv got EOF from 127.16.233.130:42231 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 16:00:28 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:28 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:28.570472 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:28.562050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:46759)
0924 16:00:28.562650 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:28.562665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:28.562831 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:28.563195 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:28.563199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:28.563616 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:28.563785 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:28.563794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.564915 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.564921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:28.565839 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:28.565851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.566675 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.566681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:28.566855 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:28.567432 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:28.567461 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:28.570167 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":347,"threads_started":1}
W20260924 16:00:28.570899 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:28.562129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:46759 (local address 127.16.233.190:42479)
0924 16:00:28.562401 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:28.562409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:28.562746 (+ 337us) server_negotiation.cc:407] Connection header received
0924 16:00:28.562968 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:28.562973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:28.563064 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:28.563162 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:28.563934 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.564767 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.565979 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.566491 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.566621 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:28.570456 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:46759: BlockingRecv error: recv got EOF from 127.16.233.131:46759 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 16:00:28 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:28 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:28.644116 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:28.635637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:47847)
0924 16:00:28.636093 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:28.636110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:28.636231 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:28.636568 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:28.636572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:28.636956 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:28.637124 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:28.637133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.638116 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.638120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:28.639106 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:28.639120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.639784 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.639788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:28.639906 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:28.640551 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:28.640570 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:28.643844 (+ 3274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":240,"threads_started":1}
W20260924 16:00:28.644268 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:28.635742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:47847 (local address 127.16.233.190:42479)
0924 16:00:28.635989 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:28.635995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:28.636194 (+ 199us) server_negotiation.cc:407] Connection header received
0924 16:00:28.636334 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:28.636337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:28.636398 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:28.636479 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:28.637282 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.637988 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.639231 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:28.639653 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:28.639745 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:28.644029 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:47847: BlockingRecv error: recv got EOF from 127.16.233.129:47847 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 24 16:00:29 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:29 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:29.094573 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:29.085133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:55917)
0924 16:00:29.085694 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:29.085711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:29.085855 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:29.086223 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:29.086226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:29.086651 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:29.086830 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:29.086839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.088021 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.088026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:29.089418 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:29.089431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.090136 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.090140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:29.090268 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:29.090874 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:29.090890 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:29.094268 (+ 3378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":321,"threads_started":1}
W20260924 16:00:29.094736 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:29.085327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:55917 (local address 127.16.233.190:42479)
0924 16:00:29.085500 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:29.085505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:29.085802 (+ 297us) server_negotiation.cc:407] Connection header received
0924 16:00:29.086006 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:29.086011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:29.086085 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:29.086188 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:29.086962 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.087867 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.089570 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.089996 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.090113 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:29.094531 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:55917: BlockingRecv error: recv got EOF from 127.16.233.130:55917 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 16:00:29 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:29 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:29.597699 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:29.571524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:60565)
0924 16:00:29.572021 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:29.572036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:29.572213 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:29.572578 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:29.572583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:29.573000 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:29.573163 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:29.573172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.574197 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.574204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:29.575231 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:29.575244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.576005 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.576011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:29.576194 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:29.594336 (+ 18142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:29.594367 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:29.597445 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":258,"threads_started":1}
W20260924 16:00:29.597901 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:29.571551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:60565 (local address 127.16.233.190:42479)
0924 16:00:29.571818 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:29.571826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:29.572125 (+ 299us) server_negotiation.cc:407] Connection header received
0924 16:00:29.572375 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:29.572379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:29.572442 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:29.572547 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:29.573325 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.574062 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.575394 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.575831 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.575973 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:29.597667 (+ 21694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:60565: BlockingRecv error: recv got EOF from 127.16.233.131:60565 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 16:00:29 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:29 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:29.653327 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:29.645010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:36257)
0924 16:00:29.645556 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:29.645572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:29.645680 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:29.645980 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:29.645983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:29.646436 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:29.646594 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:29.646602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.647585 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.647589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:29.648503 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:29.648513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.649196 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.649200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:29.649370 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:29.649986 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:29.650003 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:29.652989 (+ 2986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":320,"threads_started":1}
W20260924 16:00:29.653472 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:29.645182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:36257 (local address 127.16.233.190:42479)
0924 16:00:29.645366 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:29.645370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:29.645651 (+ 281us) server_negotiation.cc:407] Connection header received
0924 16:00:29.645780 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:29.645784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:29.645857 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:29.645941 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:29.646735 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.647474 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.648623 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:29.649038 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:29.649184 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:29.653169 (+ 3985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:36257: BlockingRecv error: recv got EOF from 127.16.233.129:36257 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 16:00:30 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:30 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:30.104240 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:30.095692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:36619)
0924 16:00:30.096175 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:30.096190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:30.096362 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:30.096693 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:30.096698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:30.097136 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:30.097290 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:30.097299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.098376 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.098381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:30.099395 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:30.099407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.100172 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.100178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:30.100366 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:30.100937 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:30.100953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:30.103918 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":267,"threads_started":1}
W20260924 16:00:30.104347 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:30.095770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:36619 (local address 127.16.233.190:42479)
0924 16:00:30.096013 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:30.096021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:30.096280 (+ 259us) server_negotiation.cc:407] Connection header received
0924 16:00:30.096496 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:30.096501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:30.096561 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:30.096676 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:30.097445 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.098179 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.099541 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.100002 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.100158 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:30.104121 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:36619: BlockingRecv error: recv got EOF from 127.16.233.130:36619 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 16:00:30 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:30 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:30.607218 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:30.598775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:50445)
0924 16:00:30.599578 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:30.599595 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:30.599700 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:30.600010 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:30.600013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:30.600397 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:30.600541 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:30.600549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.601538 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.601541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:30.602481 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:30.602492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.603164 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.603168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:30.603275 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:30.603846 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:30.603863 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:30.606893 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":532,"threads_started":1}
W20260924 16:00:30.607386 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:30.599005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:50445 (local address 127.16.233.190:42479)
0924 16:00:30.599354 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:30.599362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:30.599695 (+ 333us) server_negotiation.cc:407] Connection header received
0924 16:00:30.599821 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:30.599824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:30.599887 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:30.599972 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:30.600694 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.601408 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.602594 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.603021 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.603138 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:30.607099 (+ 3961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:50445: BlockingRecv error: recv got EOF from 127.16.233.131:50445 (error 108)
Metrics: {"server-negotiator.queue_time_us":198}
Sep 24 16:00:30 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:30 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:30.662910 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:30.654368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:36849)
0924 16:00:30.654955 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:30.654971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:30.655093 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:30.655399 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:30.655402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:30.655815 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:30.655992 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:30.656000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.656972 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.656976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:30.657981 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:30.657995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.658986 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.658992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:30.659175 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:30.659801 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:30.659823 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:30.662670 (+ 2847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":333,"threads_started":1}
W20260924 16:00:30.663184 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:30.654476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:36849 (local address 127.16.233.190:42479)
0924 16:00:30.654665 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:30.654669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:30.655051 (+ 382us) server_negotiation.cc:407] Connection header received
0924 16:00:30.655196 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:30.655200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:30.655265 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:30.655353 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:30.656115 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.656842 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.658173 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:30.658736 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:30.658969 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:30.662880 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:36849: BlockingRecv error: recv got EOF from 127.16.233.129:36849 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 16:00:31 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:31 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:31.114046 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:31.105242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:57325)
0924 16:00:31.105788 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:31.105802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:31.105957 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:31.106295 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:31.106326 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:31.106764 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:31.106944 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:31.106953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.108378 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.108382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:31.109305 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:31.109317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.109985 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.109989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:31.110114 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:31.110701 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:31.110719 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:31.113649 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":289,"threads_started":1}
W20260924 16:00:31.114095 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:31.105351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:57325 (local address 127.16.233.190:42479)
0924 16:00:31.105608 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:31.105617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:31.105884 (+ 267us) server_negotiation.cc:407] Connection header received
0924 16:00:31.106079 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:31.106084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:31.106156 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:31.106260 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:31.107088 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.108197 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.109444 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.109854 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.109950 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:31.113835 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:57325: BlockingRecv error: recv got EOF from 127.16.233.130:57325 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 16:00:31 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:31 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:31.617152 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:31.608253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:52473)
0924 16:00:31.608821 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:31.608841 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:31.608967 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:31.609320 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:31.609323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:31.609720 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:31.609904 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:31.609912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.611549 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.611554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:31.612646 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:31.612660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.613340 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.613344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:31.613462 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:31.614047 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:31.614065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:31.616903 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":333,"threads_started":1}
W20260924 16:00:31.617420 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:31.608428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:52473 (local address 127.16.233.190:42479)
0924 16:00:31.608639 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:31.608644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:31.608963 (+ 319us) server_negotiation.cc:407] Connection header received
0924 16:00:31.609077 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:31.609081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:31.609167 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:31.609260 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:31.610067 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.611262 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.612787 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.613198 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.613306 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:31.617137 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:52473: BlockingRecv error: recv got EOF from 127.16.233.131:52473 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 16:00:31 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:31 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:31.672683 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:31.663876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:51717)
0924 16:00:31.664488 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:31.664504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:31.664619 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:31.664914 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:31.664917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:31.665336 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:31.665493 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:31.665501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.666583 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.666589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:31.667544 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:31.667557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.668275 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.668279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:31.668414 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:31.669057 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:31.669114 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:31.672340 (+ 3226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":384,"threads_started":1}
W20260924 16:00:31.672771 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:31.663919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:51717 (local address 127.16.233.190:42479)
0924 16:00:31.664084 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:31.664089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:31.664578 (+ 489us) server_negotiation.cc:407] Connection header received
0924 16:00:31.664708 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:31.664712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:31.664772 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:31.664861 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:31.665615 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.666393 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.667673 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:31.668138 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:31.668232 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:31.672548 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:51717: BlockingRecv error: recv got EOF from 127.16.233.129:51717 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 16:00:32 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:32 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:00:32.127444 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:32.115125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:41433 (local address 127.16.233.190:42479)
0924 16:00:32.115370 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:32.115377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:32.115616 (+ 239us) server_negotiation.cc:407] Connection header received
0924 16:00:32.115769 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:32.115772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:32.115832 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:32.115910 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:32.116819 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.118128 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.120319 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.121150 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.121280 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:32.127163 (+ 5883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:41433: BlockingRecv error: recv got EOF from 127.16.233.130:41433 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:00:32.127806 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:32.115040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:41433)
0924 16:00:32.115521 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:32.115537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:32.115667 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:32.115980 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:32.115984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:32.116396 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:32.116632 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:32.116643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.118367 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.118374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:32.120136 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:32.120152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.121600 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.121608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:32.121863 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:32.122708 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:32.122737 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:32.127443 (+ 4706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":271,"threads_started":1}
Sep 24 16:00:32 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:32 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:32.626048 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:32.618112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:39715)
0924 16:00:32.618664 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:32.618680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:32.618802 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:32.619105 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:32.619108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:32.619469 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:32.619624 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:32.619632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.620619 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.620623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:32.621510 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:32.621521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.622223 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.622227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:32.622424 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:32.623012 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:32.623030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:32.625821 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":351,"threads_started":1}
W20260924 16:00:32.626355 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:32.618421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:39715 (local address 127.16.233.190:42479)
0924 16:00:32.618585 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:32.618592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:32.618756 (+ 164us) server_negotiation.cc:407] Connection header received
0924 16:00:32.618891 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:32.618894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:32.618968 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:32.619053 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:32.619765 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.620494 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.621663 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.622070 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.622205 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:32.626030 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:39715: BlockingRecv error: recv got EOF from 127.16.233.131:39715 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 16:00:32 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:32 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:32.682704 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:32.673742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:35981)
0924 16:00:32.674257 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:32.674274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:32.674443 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:32.674721 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:32.674724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:32.675141 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:32.675355 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:32.675376 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.676577 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.676582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:32.677766 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:32.677780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.678686 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.678692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:32.678862 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:32.679404 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:32.679421 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:32.682331 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":258,"threads_started":1}
W20260924 16:00:32.682741 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:32.673797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:35981 (local address 127.16.233.190:42479)
0924 16:00:32.674021 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:32.674031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:32.674396 (+ 365us) server_negotiation.cc:407] Connection header received
0924 16:00:32.674530 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:32.674534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:32.674596 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:32.674675 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:32.675513 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.676443 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.677909 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:32.678468 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:32.678638 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:32.682525 (+ 3887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:35981: BlockingRecv error: recv got EOF from 127.16.233.129:35981 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 16:00:33 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:33 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:33.137398 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:33.128793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:33505)
0924 16:00:33.129341 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:33.129356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:33.129531 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:33.129878 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:33.129883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:33.130436 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:33.130631 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:33.130641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.131813 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.131819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:33.132727 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:33.132738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.133467 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.133474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:33.133648 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:33.134279 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:33.134317 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:33.137055 (+ 2738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":274,"threads_started":1}
W20260924 16:00:33.137495 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:33.128954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:33505 (local address 127.16.233.190:42479)
0924 16:00:33.129181 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:33.129189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:33.129464 (+ 275us) server_negotiation.cc:407] Connection header received
0924 16:00:33.129672 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:33.129676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:33.129734 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:33.129838 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:33.130778 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.131669 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.132879 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.133283 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.133432 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:33.137248 (+ 3816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:33505: BlockingRecv error: recv got EOF from 127.16.233.130:33505 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 16:00:33 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:33 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:33.635655 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:33.627122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:36305)
0924 16:00:33.627703 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:33.627718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:33.627881 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:33.628324 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:33.628329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:33.628949 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:33.629153 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:33.629163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.630221 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.630224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:33.631159 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:33.631170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.631866 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.631870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:33.631977 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:33.632516 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:33.632534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:33.635291 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":317,"threads_started":1}
W20260924 16:00:33.635793 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:33.627221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:36305 (local address 127.16.233.190:42479)
0924 16:00:33.627400 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:33.627405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:33.627807 (+ 402us) server_negotiation.cc:407] Connection header received
0924 16:00:33.628010 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:33.628014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:33.628160 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:33.628269 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:33.629358 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.630085 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.631293 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.631705 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.631811 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:33.635513 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:36305: BlockingRecv error: recv got EOF from 127.16.233.131:36305 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 16:00:33 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:33 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:33.692173 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:33.683701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:35629)
0924 16:00:33.684198 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:33.684214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:33.684363 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:33.684706 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:33.684711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:33.685123 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:33.685296 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:33.685305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.686370 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.686376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:33.687603 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:33.687618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.688327 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.688332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:33.688497 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:33.689068 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:33.689086 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:33.691782 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":264,"threads_started":1}
W20260924 16:00:33.692222 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:33.683799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:35629 (local address 127.16.233.190:42479)
0924 16:00:33.684095 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:33.684103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:33.684309 (+ 206us) server_negotiation.cc:407] Connection header received
0924 16:00:33.684497 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:33.684502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:33.684571 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:33.684673 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:33.685443 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.686193 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.687734 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:33.688155 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:33.688317 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:33.691976 (+ 3659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:35629: BlockingRecv error: recv got EOF from 127.16.233.129:35629 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
Sep 24 16:00:34 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:34 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:34.146472 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:34.138496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:40701)
0924 16:00:34.138967 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:34.138983 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:34.139099 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:34.139399 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:34.139402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:34.139773 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:34.139946 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:34.139954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.140927 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.140931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:34.141852 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:34.141863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.142643 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.142647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:34.142774 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:34.143357 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:34.143373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:34.146130 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":264,"threads_started":1}
W20260924 16:00:34.146608 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:34.138572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:40701 (local address 127.16.233.190:42479)
0924 16:00:34.138824 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:34.138832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:34.139071 (+ 239us) server_negotiation.cc:407] Connection header received
0924 16:00:34.139201 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:34.139205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:34.139268 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:34.139350 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:34.140068 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.140800 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.141992 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.142498 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.142610 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:34.146412 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:40701: BlockingRecv error: recv got EOF from 127.16.233.130:40701 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 16:00:34 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:34 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.131@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:34.656548 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:34.636694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.131:37851)
0924 16:00:34.637273 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:34.637288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:34.637442 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:34.637827 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:34.637831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:34.638258 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:34.638458 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:34.638467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.639621 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.639630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:34.640872 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:34.640885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.641611 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.641617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:34.641791 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:34.653326 (+ 11535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:34.653348 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:34.656167 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":310,"threads_started":1}
W20260924 16:00:34.656634 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:34.636863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:37851 (local address 127.16.233.190:42479)
0924 16:00:34.637121 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:34.637128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:34.637375 (+ 247us) server_negotiation.cc:407] Connection header received
0924 16:00:34.637603 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:34.637608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:34.637684 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:34.637792 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:34.638612 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.639477 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.641013 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.641459 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.641588 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:34.656370 (+ 14782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.131:37851: BlockingRecv error: recv got EOF from 127.16.233.131:37851 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 16:00:34 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:34 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.129@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:34.708225 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:34.693312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.129:39001)
0924 16:00:34.693782 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:34.693799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:34.693928 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:34.699044 (+ 5116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:34.699053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:34.699718 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:34.699987 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:34.700011 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.701220 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.701230 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:34.703030 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:34.703050 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.703802 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.703811 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:34.703975 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:34.704684 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:34.704709 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:34.707837 (+ 3128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":217,"threads_started":1}
W20260924 16:00:34.708271 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:34.698246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:39001 (local address 127.16.233.190:42479)
0924 16:00:34.698472 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:34.698479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:34.698499 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:00:34.698559 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:34.698564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:34.698650 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:34.698793 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:34.700166 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.701056 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.703182 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:34.703640 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:34.703741 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:34.708050 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.129:39001: BlockingRecv error: recv got EOF from 127.16.233.129:39001 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 24 16:00:35 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:00:35 dist-test-slave-xn4h krb5kdc[25684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.130@KRBTEST.COM for oryx/127.16.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:00:35.156337 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:35.147948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:42479 (local address 127.16.233.130:50087)
0924 16:00:35.148547 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:35.148562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:35.148753 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:35.149093 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:35.149098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:35.149556 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:35.149727 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:35.149736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.150837 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.150844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:35.151757 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:35.151769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.152497 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.152503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:35.152686 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:35.153330 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:35.153347 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:35.156073 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.190:42479: Server oryx/127.16.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":314,"threads_started":1}
W20260924 16:00:35.156556 26428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:35.148039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:50087 (local address 127.16.233.190:42479)
0924 16:00:35.148306 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:35.148315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:35.148677 (+ 362us) server_negotiation.cc:407] Connection header received
0924 16:00:35.148877 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:35.148884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:35.148953 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:35.149059 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:35.149872 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.150693 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.151896 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.152343 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.152474 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:35.156304 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.130:50087: BlockingRecv error: recv got EOF from 127.16.233.130:50087 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 16:00:35.455127 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26278
I20260924 16:00:35.468199 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:35.456195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.16.233.129:36299)
0924 16:00:35.456784 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:35.456791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:35.456819 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:00:35.456878 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:35.456884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:35.457120 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:35.457276 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:35.458831 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.459946 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.461142 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.461703 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.461812 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:35.464418 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:35.464428 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:35.464443 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:35.464487 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:35.467649 (+ 3162us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.233.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 16:00:35.467956 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43798: Request ticket server kudu/127.16.233.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":300,"threads_started":1}
W20260924 16:00:35.468427 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26278: Not authorized: Client connection negotiation failed: client connection to 127.16.233.129:36299: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.233.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:00:35.468645 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 26278
I20260924 16:00:35.486006 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26430
W20260924 16:00:35.498658 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26430: Not authorized: Client connection negotiation failed: client connection to 127.16.233.130:38899: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.233.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:00:35.498732 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:35.487179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.16.233.130:38899)
0924 16:00:35.487770 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:35.487780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:35.487812 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:00:35.487878 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:35.487884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:35.488092 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:35.488350 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:35.489683 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.490844 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.492276 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.492889 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.493035 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:35.495529 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:35.495537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:35.495549 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:35.495581 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:35.497901 (+ 2320us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.233.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 16:00:35.498247 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42420: Request ticket server kudu/127.16.233.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":302,"threads_started":1}
I20260924 16:00:35.498826 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 26430
W20260924 16:00:35.498812 26746 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42420: Request ticket server kudu/127.16.233.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:00:35.516129 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26565
I20260924 16:00:35.528369 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:35.517140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.16.233.131:38007)
0924 16:00:35.517677 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:35.517684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:35.517709 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:00:35.517781 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:35.517788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:35.518023 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:35.518227 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:35.519764 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.520830 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.522023 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:35.522658 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:35.522809 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:35.525244 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:35.525253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:35.525267 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:35.525305 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:35.527702 (+ 2397us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.233.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 16:00:35.528038 (+ 336us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49076: Request ticket server kudu/127.16.233.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":279,"threads_started":1}
W20260924 16:00:35.528407 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26565: Not authorized: Client connection negotiation failed: client connection to 127.16.233.131:38007: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.233.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 16:00:35.528489 26747 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49076: Request ticket server kudu/127.16.233.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:00:35.528589 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 26565
I20260924 16:00:35.544992 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26209
I20260924 16:00:35.546376 26271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:35.663353 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 26209
2026-09-24T16:00:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21188 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T16:00:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:00:35Z Disabled control of system clock
I20260924 16:00:35.738200 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:38239
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 16:00:36.059029 26752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:36.059345 26752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:36.059422 26752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:36.066602 26752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:00:36.066709 26752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:36.066745 26752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:36.066802 26752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:00:36.066846 26752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:00:36.077003 26752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:38239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:36.078261 26752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:36.080178 26752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:36.092180 26757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:36.092263 26758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:36.092288 26760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:36.097434 26752 server_base.cc:1034] running on GCE node
I20260924 16:00:36.098242 26752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:36.099705 26752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:36.100940 26752 hybrid_clock.cc:648] HybridClock initialized: now 1790265636100892 us; error 27 us; skew 500 ppm
I20260924 16:00:36.101334 26752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:00:36.103725 26752 webserver.cc:460] Webserver started at http://127.16.233.190:41331/ using document root <none> and password file <none>
I20260924 16:00:36.104415 26752 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:36.104530 26752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:36.104894 26752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:36.107870 26752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0"
format_stamp: "Formatted at 2026-09-24 16:00:36 on dist-test-slave-xn4h"
server_key: "32a692db34f5a8a8a4ce76bd05633ea1"
server_key_iv: "6341d1da1d09f4a5c18a35be10cc17e0"
server_key_version: "encryptionkey@0"
I20260924 16:00:36.108667 26752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0"
format_stamp: "Formatted at 2026-09-24 16:00:36 on dist-test-slave-xn4h"
server_key: "32a692db34f5a8a8a4ce76bd05633ea1"
server_key_iv: "6341d1da1d09f4a5c18a35be10cc17e0"
server_key_version: "encryptionkey@0"
I20260924 16:00:36.113569 26752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:00:36.117218 26766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:36.118877 26752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:36.119060 26752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0"
format_stamp: "Formatted at 2026-09-24 16:00:36 on dist-test-slave-xn4h"
server_key: "32a692db34f5a8a8a4ce76bd05633ea1"
server_key_iv: "6341d1da1d09f4a5c18a35be10cc17e0"
server_key_version: "encryptionkey@0"
I20260924 16:00:36.119334 26752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:36.142874 26752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:36.143705 26752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:36.143996 26752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:36.162456 26752 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:38239
I20260924 16:00:36.162462 26818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:38239 every 8 connection(s)
I20260924 16:00:36.163755 26752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:00:36.167330 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 26752
I20260924 16:00:36.167497 26819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:36.167539 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:00:36.168239 17318 external_mini_cluster.cc:1428] Setting key 188cb8f11edf82828ee45c972f49148b
I20260924 16:00:36.181937 26819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0: Bootstrap starting.
I20260924 16:00:36.184286 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:36.170234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.16.233.190:38239)
0924 16:00:36.171411 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:36.171431 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:36.171485 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:00:36.174255 (+ 2770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:36.174290 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:36.174872 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:36.175311 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:36.175825 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:36.177676 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:36.179221 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:36.180265 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:36.180445 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:36.181589 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:36.181601 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:36.181618 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:36.182265 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:36.182420 (+ 155us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:36.182447 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:36.182563 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:36.182743 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 16:00:36.183040 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":374,"threads_started":1}
I20260924 16:00:36.185570 26819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:36.186658 26819 log.cc:824] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:36.189150 26819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0: No bootstrap required, opened a new log
I20260924 16:00:36.195842 26819 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38239 } }
I20260924 16:00:36.196203 26819 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:36.196343 26819 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7c1c8155aa4b1c98ccdb7e672197a0, State: Initialized, Role: FOLLOWER
I20260924 16:00:36.196889 26819 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38239 } }
I20260924 16:00:36.197119 26819 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:00:36.197258 26819 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:00:36.197432 26819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:36.199810 26819 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38239 } }
I20260924 16:00:36.200294 26819 leader_election.cc:302] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7c1c8155aa4b1c98ccdb7e672197a0; no voters:
I20260924 16:00:36.200865 26819 leader_election.cc:288] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:00:36.201031 26824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:36.201833 26824 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [term 1 LEADER]: Becoming Leader. State: Replica: cd7c1c8155aa4b1c98ccdb7e672197a0, State: Running, Role: LEADER
I20260924 16:00:36.202448 26824 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38239 } }
I20260924 16:00:36.203018 26819 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:00:36.205144 26825 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38239 } } }
I20260924 16:00:36.205835 26826 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7c1c8155aa4b1c98ccdb7e672197a0. Latest consensus state: current_term: 1 leader_uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c1c8155aa4b1c98ccdb7e672197a0" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 38239 } } }
I20260924 16:00:36.206480 26826 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:36.206650 26825 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:36.208274 26833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:00:36.213639 26833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:00:36.221418 26833 catalog_manager.cc:1347] Generated new cluster ID: 50e0f6450ee440bea01594d23b3d0736
I20260924 16:00:36.221516 26833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:00:36.247946 26833 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:00:36.249267 26833 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:00:36.254693 26833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7c1c8155aa4b1c98ccdb7e672197a0: Generated new TSK 0
I20260924 16:00:36.255456 26833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:00:36.268728 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38239
--builtin_ntp_servers=127.16.233.148:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 16:00:36.595777 26843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:36.596016 26843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:36.596194 26843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:36.603356 26843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:36.603440 26843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:36.603627 26843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:00:36.612715 26843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:36.613950 26843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:36.615790 26843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:36.629015 26848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:36.629056 26849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:36.629237 26851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:36.629952 26843 server_base.cc:1034] running on GCE node
I20260924 16:00:36.630638 26843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:36.631517 26843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:36.632759 26843 hybrid_clock.cc:648] HybridClock initialized: now 1790265636632706 us; error 62 us; skew 500 ppm
I20260924 16:00:36.633077 26843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:00:36.635403 26843 webserver.cc:460] Webserver started at http://127.16.233.129:33127/ using document root <none> and password file <none>
I20260924 16:00:36.636112 26843 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:36.636253 26843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:36.636584 26843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:36.639501 26843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "b7810740e068467badb5c46019d913c1"
format_stamp: "Formatted at 2026-09-24 16:00:36 on dist-test-slave-xn4h"
server_key: "29b3a049209f6d98e199aab58bb7c3a2"
server_key_iv: "af676f4aa6f31ea5287a479d58c2894f"
server_key_version: "encryptionkey@0"
I20260924 16:00:36.640321 26843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "b7810740e068467badb5c46019d913c1"
format_stamp: "Formatted at 2026-09-24 16:00:36 on dist-test-slave-xn4h"
server_key: "29b3a049209f6d98e199aab58bb7c3a2"
server_key_iv: "af676f4aa6f31ea5287a479d58c2894f"
server_key_version: "encryptionkey@0"
I20260924 16:00:36.645334 26843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 16:00:36.648602 26857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:36.649901 26843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:00:36.650079 26843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "b7810740e068467badb5c46019d913c1"
format_stamp: "Formatted at 2026-09-24 16:00:36 on dist-test-slave-xn4h"
server_key: "29b3a049209f6d98e199aab58bb7c3a2"
server_key_iv: "af676f4aa6f31ea5287a479d58c2894f"
server_key_version: "encryptionkey@0"
I20260924 16:00:36.650290 26843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:36.689725 26843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:36.690677 26843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:36.690997 26843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:36.691932 26843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:36.693349 26843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:36.693475 26843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:36.693602 26843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:36.693688 26843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:36.728077 26843 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:42895
I20260924 16:00:36.728180 26971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:42895 every 8 connection(s)
I20260924 16:00:36.729409 26843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:00:36.730665 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 26843
I20260924 16:00:36.730873 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:00:36.731783 17318 external_mini_cluster.cc:1428] Setting key 03998a630ab547b2cbb3809fa19de988
I20260924 16:00:36.734819 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38239
--builtin_ntp_servers=127.16.233.148:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:00:36.743842 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:36.732563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:58269 (local address 127.16.233.190:38239)
0924 16:00:36.733079 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:36.733086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:36.734565 (+ 1479us) server_negotiation.cc:407] Connection header received
0924 16:00:36.736517 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:36.736526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:36.736624 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:36.736746 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:36.737838 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:36.739090 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:36.740641 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:36.741202 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:36.741304 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:36.743012 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:36.743016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:36.743020 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:36.743044 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:36.743092 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:36.743106 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:36.743180 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:36.743499 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 16:00:36.743593 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
I20260924 16:00:36.745283 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:36.733269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:38239 (local address 127.16.233.129:58269)
0924 16:00:36.734396 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:36.734444 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:36.736210 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:36.736950 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:36.736958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:36.737059 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:36.737650 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:36.737666 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:36.739240 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:36.739244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:36.740482 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:36.740494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:36.741451 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:36.741458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:36.741792 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:36.742639 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:00:36.742645 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:36.742699 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:00:36.742809 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:36.743249 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:36.743263 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:36.743282 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 16:00:36.743468 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 16:00:36.743890 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":348,"threads_started":1}
I20260924 16:00:36.746840 26972 heartbeater.cc:347] Connected to a master server at 127.16.233.190:38239
I20260924 16:00:36.747170 26972 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:36.747942 26972 heartbeater.cc:511] Master 127.16.233.190:38239 requested a full tablet report, sending...
I20260924 16:00:36.750094 26783 ts_manager.cc:194] Registered new tserver with Master: b7810740e068467badb5c46019d913c1 (127.16.233.129:42895)
I20260924 16:00:36.751782 26783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.129:58269
W20260924 16:00:37.063877 26977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:37.064157 26977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:37.064296 26977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:37.071451 26977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:37.071552 26977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:37.071739 26977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:00:37.082357 26977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:37.083631 26977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:37.085454 26977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:37.098464 26982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:37.098497 26985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:37.098419 26983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:37.099880 26977 server_base.cc:1034] running on GCE node
I20260924 16:00:37.100606 26977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:37.101580 26977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:37.102820 26977 hybrid_clock.cc:648] HybridClock initialized: now 1790265637102754 us; error 54 us; skew 500 ppm
I20260924 16:00:37.103199 26977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:00:37.105500 26977 webserver.cc:460] Webserver started at http://127.16.233.130:34537/ using document root <none> and password file <none>
I20260924 16:00:37.106143 26977 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:37.106287 26977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:37.106662 26977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:37.109596 26977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "b973fffcb6aa4e78b7cf50272e8adbf9"
format_stamp: "Formatted at 2026-09-24 16:00:37 on dist-test-slave-xn4h"
server_key: "e4c4f8850ea9e1ba26f060762196536f"
server_key_iv: "ea6b0708e619ec258f6654317623b813"
server_key_version: "encryptionkey@0"
I20260924 16:00:37.110459 26977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "b973fffcb6aa4e78b7cf50272e8adbf9"
format_stamp: "Formatted at 2026-09-24 16:00:37 on dist-test-slave-xn4h"
server_key: "e4c4f8850ea9e1ba26f060762196536f"
server_key_iv: "ea6b0708e619ec258f6654317623b813"
server_key_version: "encryptionkey@0"
I20260924 16:00:37.115446 26977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:00:37.118920 26991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:37.120261 26977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:00:37.120424 26977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "b973fffcb6aa4e78b7cf50272e8adbf9"
format_stamp: "Formatted at 2026-09-24 16:00:37 on dist-test-slave-xn4h"
server_key: "e4c4f8850ea9e1ba26f060762196536f"
server_key_iv: "ea6b0708e619ec258f6654317623b813"
server_key_version: "encryptionkey@0"
I20260924 16:00:37.120612 26977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:37.139639 26977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:37.140372 26977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:37.140616 26977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:37.141460 26977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:37.142845 26977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:37.142935 26977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:37.143011 26977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:37.143066 26977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:37.176406 26977 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:37725
I20260924 16:00:37.176460 27105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:37725 every 8 connection(s)
I20260924 16:00:37.177675 26977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:00:37.179512 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 26977
I20260924 16:00:37.179697 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:00:37.180245 17318 external_mini_cluster.cc:1428] Setting key ceeed2af2483cb900cda4a5c0bbc7945
I20260924 16:00:37.183401 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38239
--builtin_ntp_servers=127.16.233.148:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:00:37.190209 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:37.180801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:55985 (local address 127.16.233.190:38239)
0924 16:00:37.180990 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:37.180996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:37.182122 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 16:00:37.183981 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:37.183987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:37.184068 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:37.184184 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:37.185183 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.185869 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.187443 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.187818 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.187947 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:37.189340 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:37.189344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:37.189347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:37.189363 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:37.189395 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:37.189407 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:37.189513 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:37.189756 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 16:00:37.189854 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:00:37.191135 27108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:37.180927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:38239 (local address 127.16.233.130:55985)
0924 16:00:37.181977 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:37.182017 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:37.183738 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:37.184410 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:37.184418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:37.184500 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:37.185026 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:37.185041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.186020 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.186026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:37.187291 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:37.187303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.188027 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.188033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:37.188248 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:37.188990 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:00:37.188993 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:37.189029 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:00:37.189107 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:37.189513 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:37.189527 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:37.189537 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:00:37.189728 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 16:00:37.190113 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":332,"threads_started":1}
I20260924 16:00:37.192565 27106 heartbeater.cc:347] Connected to a master server at 127.16.233.190:38239
I20260924 16:00:37.192873 27106 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:37.193611 27106 heartbeater.cc:511] Master 127.16.233.190:38239 requested a full tablet report, sending...
I20260924 16:00:37.195015 26783 ts_manager.cc:194] Registered new tserver with Master: b973fffcb6aa4e78b7cf50272e8adbf9 (127.16.233.130:37725)
I20260924 16:00:37.195844 26783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.130:55985
W20260924 16:00:37.506078 27109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:37.506412 27109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:37.506561 27109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:37.513826 27109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:37.513962 27109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:37.514139 27109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:00:37.523118 27109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:38239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:37.524613 27109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:37.526582 27109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:37.539613 27114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:37.539670 27115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:37.539702 27117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:37.540524 27109 server_base.cc:1034] running on GCE node
I20260924 16:00:37.541149 27109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:37.541994 27109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:37.543219 27109 hybrid_clock.cc:648] HybridClock initialized: now 1790265637543184 us; error 46 us; skew 500 ppm
I20260924 16:00:37.543529 27109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:00:37.545709 27109 webserver.cc:460] Webserver started at http://127.16.233.131:44433/ using document root <none> and password file <none>
I20260924 16:00:37.546437 27109 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:37.546568 27109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:37.546892 27109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:37.549638 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "620dcc340a3f4ce8a501bbc951f51b87"
format_stamp: "Formatted at 2026-09-24 16:00:37 on dist-test-slave-xn4h"
server_key: "bd0224989738c9f9ae1764a70ea3aa36"
server_key_iv: "28cf5392e6f0c9604d13d98e723ba992"
server_key_version: "encryptionkey@0"
I20260924 16:00:37.550491 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "620dcc340a3f4ce8a501bbc951f51b87"
format_stamp: "Formatted at 2026-09-24 16:00:37 on dist-test-slave-xn4h"
server_key: "bd0224989738c9f9ae1764a70ea3aa36"
server_key_iv: "28cf5392e6f0c9604d13d98e723ba992"
server_key_version: "encryptionkey@0"
I20260924 16:00:37.555678 27109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:00:37.559078 27123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:37.560390 27109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:00:37.560580 27109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "620dcc340a3f4ce8a501bbc951f51b87"
format_stamp: "Formatted at 2026-09-24 16:00:37 on dist-test-slave-xn4h"
server_key: "bd0224989738c9f9ae1764a70ea3aa36"
server_key_iv: "28cf5392e6f0c9604d13d98e723ba992"
server_key_version: "encryptionkey@0"
I20260924 16:00:37.560806 27109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:37.585881 27109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:37.586756 27109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:37.587044 27109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:37.587931 27109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:37.589294 27109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:37.589412 27109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:37.589535 27109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:37.589624 27109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:37.622000 27109 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:43843
I20260924 16:00:37.622051 27238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:43843 every 8 connection(s)
I20260924 16:00:37.623327 27109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:00:37.626495 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 27109
I20260924 16:00:37.626659 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:00:37.627255 17318 external_mini_cluster.cc:1428] Setting key 97280eb2bd12e3d3843d4e8d2489801c
I20260924 16:00:37.636808 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:37.626177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:52017 (local address 127.16.233.190:38239)
0924 16:00:37.626403 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:37.626413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:37.628222 (+ 1809us) server_negotiation.cc:407] Connection header received
0924 16:00:37.630644 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:37.630649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:37.630749 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:37.630850 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:37.631881 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.632608 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.634121 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.634609 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.634742 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:37.635975 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:37.635981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:37.635985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:37.636012 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:37.636043 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:37.636054 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:37.636146 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:37.636382 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 16:00:37.636478 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:00:37.637778 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:37.626955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:38239 (local address 127.16.233.131:52017)
0924 16:00:37.628046 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:37.628096 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:37.630378 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:37.631074 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:37.631082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:37.631180 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:37.631726 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:37.631742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.632763 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.632769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:37.633972 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:37.633984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.634827 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.634834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:37.635071 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:37.635678 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:00:37.635680 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:37.635717 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:00:37.635792 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:37.636145 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:37.636178 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:37.636186 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:00:37.636374 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 16:00:37.636775 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":360,"threads_started":1}
I20260924 16:00:37.639006 27239 heartbeater.cc:347] Connected to a master server at 127.16.233.190:38239
I20260924 16:00:37.639307 27239 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:37.640013 27239 heartbeater.cc:511] Master 127.16.233.190:38239 requested a full tablet report, sending...
I20260924 16:00:37.641429 26783 ts_manager.cc:194] Registered new tserver with Master: 620dcc340a3f4ce8a501bbc951f51b87 (127.16.233.131:43843)
I20260924 16:00:37.642402 26783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.131:52017
I20260924 16:00:37.642740 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:00:37.648896 26974 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:37.647169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.16.233.190:38239)
0924 16:00:37.647343 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:37.647347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:37.647956 (+ 609us) server_negotiation.cc:407] Connection header received
0924 16:00:37.648060 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:37.648064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:37.648143 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:37.648230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:37.648662 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59296: BlockingRecv error: recv got EOF from 127.0.0.1:59296 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:00:37.649681 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26843
I20260924 16:00:37.657727 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:37.650714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.16.233.129:42895)
0924 16:00:37.651277 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:37.651284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:37.651306 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:00:37.651357 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:37.651363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:37.651538 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:37.651692 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:37.652095 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.653061 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.654577 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.655209 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.655362 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:37.656358 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:37.656366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:37.656378 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:37.656941 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:37.656985 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:37.657004 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:37.657139 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:37.657256 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 16:00:37.657387 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":283,"threads_started":1}
I20260924 16:00:37.658939 26967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:37.741317 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 26843
I20260924 16:00:37.757709 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26977
I20260924 16:00:37.765396 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:37.758610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.16.233.130:37725)
0924 16:00:37.759214 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:37.759221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:37.759249 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:00:37.759315 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:37.759322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:37.759544 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:37.759691 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:37.760025 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.761032 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.762183 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.762838 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.762936 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:37.763900 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:37.763913 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:37.763930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:37.764590 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:37.764645 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:37.764665 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:37.764750 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:37.764898 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 16:00:37.765030 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":323,"threads_started":1}
I20260924 16:00:37.766616 27101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:37.847213 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 26977
I20260924 16:00:37.864082 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 27109
I20260924 16:00:37.872443 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:37.865054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.16.233.131:43843)
0924 16:00:37.865693 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:37.865704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:37.865735 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:00:37.865804 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:37.865823 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:37.866066 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:37.866238 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:37.866763 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.867822 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.869072 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:37.869628 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:37.869771 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:37.870970 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:37.870978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:37.870991 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:37.871625 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:37.871671 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:37.871692 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:37.871828 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:37.871951 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 16:00:37.872075 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":316,"threads_started":1}
I20260924 16:00:37.873662 27234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:37.955746 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 27109
I20260924 16:00:37.971796 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 26752
I20260924 16:00:37.973052 26814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:38.075769 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 26752
2026-09-24T16:00:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2408 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T16:00:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:00:38Z Disabled control of system clock
I20260924 16:00:38.150456 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:37015
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:37015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:00:38.472292 27258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:38.472633 27258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:38.472802 27258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:38.480268 27258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:00:38.480399 27258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:38.480495 27258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:38.480605 27258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:00:38.480695 27258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:00:38.489916 27258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:37015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:37015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:38.491181 27258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:38.492936 27258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:38.504505 27264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:38.504455 27265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:38.504482 27267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:38.505352 27258 server_base.cc:1034] running on GCE node
I20260924 16:00:38.506062 27258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:38.507437 27258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:38.508668 27258 hybrid_clock.cc:648] HybridClock initialized: now 1790265638508626 us; error 55 us; skew 500 ppm
I20260924 16:00:38.508997 27258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:00:38.511313 27258 webserver.cc:460] Webserver started at http://127.16.233.190:35411/ using document root <none> and password file <none>
I20260924 16:00:38.511996 27258 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:38.512145 27258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:38.512470 27258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:38.516711 27258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "1d7981c85582471f81339b8fc269389c"
format_stamp: "Formatted at 2026-09-24 16:00:38 on dist-test-slave-xn4h"
server_key: "d6fcf03860e9a653363d0994bbbe8b3f"
server_key_iv: "6afb90c6f5443a5220be0dc461ea7adb"
server_key_version: "encryptionkey@0"
I20260924 16:00:38.517526 27258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "1d7981c85582471f81339b8fc269389c"
format_stamp: "Formatted at 2026-09-24 16:00:38 on dist-test-slave-xn4h"
server_key: "d6fcf03860e9a653363d0994bbbe8b3f"
server_key_iv: "6afb90c6f5443a5220be0dc461ea7adb"
server_key_version: "encryptionkey@0"
I20260924 16:00:38.522727 27258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:00:38.526115 27273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:38.527438 27258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:38.527619 27258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "1d7981c85582471f81339b8fc269389c"
format_stamp: "Formatted at 2026-09-24 16:00:38 on dist-test-slave-xn4h"
server_key: "d6fcf03860e9a653363d0994bbbe8b3f"
server_key_iv: "6afb90c6f5443a5220be0dc461ea7adb"
server_key_version: "encryptionkey@0"
I20260924 16:00:38.527861 27258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:38.533440 27258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:38.534153 27258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:38.534483 27258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:38.552506 27258 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:37015
I20260924 16:00:38.552538 27324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:37015 every 8 connection(s)
I20260924 16:00:38.553860 27258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:00:38.557492 27325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:38.560525 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 27258
I20260924 16:00:38.560737 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:00:38.561457 17318 external_mini_cluster.cc:1428] Setting key fcd6da124ac38c791c1723be9194a115
I20260924 16:00:38.569722 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:38.563591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.16.233.190:37015)
0924 16:00:38.564585 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:38.564595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:38.564636 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:00:38.566185 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:38.566209 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:38.566700 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:38.567039 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:38.568140 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:38.568146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:38.568161 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:38.568175 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:38.568234 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:38.568250 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:38.568447 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:38.568567 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 16:00:38.568853 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":374,"threads_started":1}
I20260924 16:00:38.570266 27325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c: Bootstrap starting.
I20260924 16:00:38.573225 27325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:38.574361 27325 log.cc:824] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:38.576844 27325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c: No bootstrap required, opened a new log
I20260924 16:00:38.583948 27325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7981c85582471f81339b8fc269389c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37015 } }
I20260924 16:00:38.584309 27325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:38.584415 27325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7981c85582471f81339b8fc269389c, State: Initialized, Role: FOLLOWER
I20260924 16:00:38.584939 27325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7981c85582471f81339b8fc269389c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37015 } }
I20260924 16:00:38.585175 27325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:00:38.585321 27325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:00:38.585481 27325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:38.587610 27325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7981c85582471f81339b8fc269389c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37015 } }
I20260924 16:00:38.588084 27325 leader_election.cc:302] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d7981c85582471f81339b8fc269389c; no voters:
I20260924 16:00:38.588683 27325 leader_election.cc:288] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:00:38.588828 27330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:38.589608 27330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [term 1 LEADER]: Becoming Leader. State: Replica: 1d7981c85582471f81339b8fc269389c, State: Running, Role: LEADER
I20260924 16:00:38.590169 27330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7981c85582471f81339b8fc269389c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37015 } }
I20260924 16:00:38.590830 27325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:00:38.592969 27331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d7981c85582471f81339b8fc269389c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7981c85582471f81339b8fc269389c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37015 } } }
I20260924 16:00:38.593463 27331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:38.593248 27332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d7981c85582471f81339b8fc269389c. Latest consensus state: current_term: 1 leader_uuid: "1d7981c85582471f81339b8fc269389c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7981c85582471f81339b8fc269389c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37015 } } }
I20260924 16:00:38.593647 27332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c [sys.catalog]: This master's current role is: LEADER
W20260924 16:00:38.602161 27346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:00:38.602326 27346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:00:38.602499 27336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:00:38.604142 27336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:00:38.611797 27336 catalog_manager.cc:1347] Generated new cluster ID: 9e5f968e65fc44d69499b03ce3ec4a05
I20260924 16:00:38.611971 27336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:00:38.639564 27336 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:00:38.640069 27336 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:00:38.652149 27336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d7981c85582471f81339b8fc269389c: Generated new TSK 0
I20260924 16:00:38.653162 27336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:00:38.663844 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37015
--builtin_ntp_servers=127.16.233.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:00:38.997239 27349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:38.997550 27349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:38.997722 27349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:39.005537 27349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:39.005687 27349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:39.005851 27349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:00:39.015156 27349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:39.016573 27349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:39.018564 27349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:39.031675 27355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:39.031883 27358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:39.031647 27354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:39.032454 27349 server_base.cc:1034] running on GCE node
I20260924 16:00:39.033030 27349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:39.033969 27349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:39.035197 27349 hybrid_clock.cc:648] HybridClock initialized: now 1790265639035136 us; error 43 us; skew 500 ppm
I20260924 16:00:39.035563 27349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:00:39.037887 27349 webserver.cc:460] Webserver started at http://127.16.233.129:37737/ using document root <none> and password file <none>
I20260924 16:00:39.038695 27349 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:39.038815 27349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:39.039152 27349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:39.041989 27349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "9229da37f701469e90187630481f71ab"
format_stamp: "Formatted at 2026-09-24 16:00:39 on dist-test-slave-xn4h"
server_key: "b89e65bc0883571c7380d2937a59e42f"
server_key_iv: "8143ad9d1a954c1638e79f6d930adc02"
server_key_version: "encryptionkey@0"
I20260924 16:00:39.042806 27349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "9229da37f701469e90187630481f71ab"
format_stamp: "Formatted at 2026-09-24 16:00:39 on dist-test-slave-xn4h"
server_key: "b89e65bc0883571c7380d2937a59e42f"
server_key_iv: "8143ad9d1a954c1638e79f6d930adc02"
server_key_version: "encryptionkey@0"
I20260924 16:00:39.047971 27349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:00:39.051482 27364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:39.052878 27349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:00:39.053057 27349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "9229da37f701469e90187630481f71ab"
format_stamp: "Formatted at 2026-09-24 16:00:39 on dist-test-slave-xn4h"
server_key: "b89e65bc0883571c7380d2937a59e42f"
server_key_iv: "8143ad9d1a954c1638e79f6d930adc02"
server_key_version: "encryptionkey@0"
I20260924 16:00:39.053290 27349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:39.058961 27349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:39.059762 27349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:39.060031 27349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:39.060952 27349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:39.062404 27349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:39.062513 27349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:39.062649 27349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:39.062753 27349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:39.096900 27349 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:38193
I20260924 16:00:39.096982 27477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:38193 every 8 connection(s)
I20260924 16:00:39.098258 27349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:00:39.106865 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:39.101014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:34087 (local address 127.16.233.190:37015)
0924 16:00:39.101455 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:39.101460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:39.102797 (+ 1337us) server_negotiation.cc:407] Connection header received
0924 16:00:39.104514 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:39.104520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:39.104575 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:39.104680 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:39.106121 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:39.106126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:39.106129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:39.106146 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:39.106189 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:39.106202 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:39.106364 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:39.106554 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 16:00:39.106616 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":235,"threads_started":1}
I20260924 16:00:39.107879 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:39.101661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:37015 (local address 127.16.233.129:34087)
0924 16:00:39.102626 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:39.102672 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:39.104220 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:39.104893 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:39.104902 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:39.105000 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:39.105805 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:00:39.105807 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:39.105848 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:00:39.105942 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:39.106351 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:39.106368 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:39.106375 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:00:39.106608 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 16:00:39.106825 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":250,"threads_started":1}
I20260924 16:00:39.108512 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 27349
I20260924 16:00:39.108764 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:00:39.109500 17318 external_mini_cluster.cc:1428] Setting key 92b44f9622a97d3659aaf8b95073ce05
I20260924 16:00:39.109769 27478 heartbeater.cc:347] Connected to a master server at 127.16.233.190:37015
I20260924 16:00:39.110101 27478 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:39.110867 27478 heartbeater.cc:511] Master 127.16.233.190:37015 requested a full tablet report, sending...
I20260924 16:00:39.112543 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37015
--builtin_ntp_servers=127.16.233.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 16:00:39.113322 27290 ts_manager.cc:194] Registered new tserver with Master: 9229da37f701469e90187630481f71ab (127.16.233.129:38193)
W20260924 16:00:39.435288 27482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:39.435566 27482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:39.435667 27482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:39.442903 27482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:39.442999 27482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:39.443147 27482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:00:39.452090 27482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:39.453369 27482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:39.455178 27482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:39.468817 27488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:39.469048 27487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:39.469175 27491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:39.469808 27482 server_base.cc:1034] running on GCE node
I20260924 16:00:39.470608 27482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:39.471628 27482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:39.472898 27482 hybrid_clock.cc:648] HybridClock initialized: now 1790265639472827 us; error 61 us; skew 500 ppm
I20260924 16:00:39.473276 27482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:00:39.475677 27482 webserver.cc:460] Webserver started at http://127.16.233.130:41097/ using document root <none> and password file <none>
I20260924 16:00:39.476421 27482 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:39.476560 27482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:39.476907 27482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:39.479951 27482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "8fd895ce4dce43a2b14868c38bb059d2"
format_stamp: "Formatted at 2026-09-24 16:00:39 on dist-test-slave-xn4h"
server_key: "7425672aacfd7078b6cdaadeb7790bc8"
server_key_iv: "820ed641d3c5b9cfa9c14b1bddebe248"
server_key_version: "encryptionkey@0"
I20260924 16:00:39.480798 27482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "8fd895ce4dce43a2b14868c38bb059d2"
format_stamp: "Formatted at 2026-09-24 16:00:39 on dist-test-slave-xn4h"
server_key: "7425672aacfd7078b6cdaadeb7790bc8"
server_key_iv: "820ed641d3c5b9cfa9c14b1bddebe248"
server_key_version: "encryptionkey@0"
I20260924 16:00:39.486137 27482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:00:39.489651 27497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:39.491082 27482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:39.491284 27482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "8fd895ce4dce43a2b14868c38bb059d2"
format_stamp: "Formatted at 2026-09-24 16:00:39 on dist-test-slave-xn4h"
server_key: "7425672aacfd7078b6cdaadeb7790bc8"
server_key_iv: "820ed641d3c5b9cfa9c14b1bddebe248"
server_key_version: "encryptionkey@0"
I20260924 16:00:39.491523 27482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:39.497527 27482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:39.498436 27482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:39.498723 27482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:39.499675 27482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:39.501332 27482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:39.501456 27482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:39.501601 27482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:39.501690 27482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:39.535580 27482 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:40891
I20260924 16:00:39.535669 27610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:40891 every 8 connection(s)
I20260924 16:00:39.536942 27482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:00:39.545178 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:39.539600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:54287 (local address 127.16.233.190:37015)
0924 16:00:39.539830 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:39.539837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:39.541248 (+ 1411us) server_negotiation.cc:407] Connection header received
0924 16:00:39.542958 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:39.542962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:39.543027 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:39.543120 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:39.544431 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:39.544436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:39.544439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:39.544456 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:39.544488 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:39.544500 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:39.544597 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:39.544765 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 16:00:39.544818 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:00:39.545279 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 27482
I20260924 16:00:39.545478 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:00:39.545951 17318 external_mini_cluster.cc:1428] Setting key 5e0f4d0086d75a529ce780f49d5321e2
I20260924 16:00:39.546070 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:39.540203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:37015 (local address 127.16.233.130:54287)
0924 16:00:39.541095 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:39.541137 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:39.542712 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:39.543322 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:39.543330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:39.543418 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:39.544119 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:00:39.544121 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:39.544158 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:00:39.544245 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:39.544589 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:39.544597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:39.544602 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:00:39.544807 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 16:00:39.544997 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":283,"threads_started":1}
I20260924 16:00:39.547744 27611 heartbeater.cc:347] Connected to a master server at 127.16.233.190:37015
I20260924 16:00:39.548096 27611 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:39.548807 27611 heartbeater.cc:511] Master 127.16.233.190:37015 requested a full tablet report, sending...
I20260924 16:00:39.549017 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37015
--builtin_ntp_servers=127.16.233.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 16:00:39.550472 27290 ts_manager.cc:194] Registered new tserver with Master: 8fd895ce4dce43a2b14868c38bb059d2 (127.16.233.130:40891)
W20260924 16:00:39.877470 27614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:39.877737 27614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:39.877822 27614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:39.885031 27614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:39.885133 27614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:39.885238 27614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:00:39.894577 27614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:39.895960 27614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:39.897863 27614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:39.910645 27622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:39.910698 27619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:39.910720 27620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:39.911794 27614 server_base.cc:1034] running on GCE node
I20260924 16:00:39.912336 27614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:39.913173 27614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:39.914434 27614 hybrid_clock.cc:648] HybridClock initialized: now 1790265639914382 us; error 37 us; skew 500 ppm
I20260924 16:00:39.914763 27614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:00:39.924280 27614 webserver.cc:460] Webserver started at http://127.16.233.131:42803/ using document root <none> and password file <none>
I20260924 16:00:39.925014 27614 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:39.925135 27614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:39.925467 27614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:39.928287 27614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "e8631d30b519410ebead93cd4ffe5a0e"
format_stamp: "Formatted at 2026-09-24 16:00:39 on dist-test-slave-xn4h"
server_key: "27d5316ce0584b2a8a8ed328b7d3293e"
server_key_iv: "cedb037c9c6fa13414452b98a316dffd"
server_key_version: "encryptionkey@0"
I20260924 16:00:39.929051 27614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "e8631d30b519410ebead93cd4ffe5a0e"
format_stamp: "Formatted at 2026-09-24 16:00:39 on dist-test-slave-xn4h"
server_key: "27d5316ce0584b2a8a8ed328b7d3293e"
server_key_iv: "cedb037c9c6fa13414452b98a316dffd"
server_key_version: "encryptionkey@0"
I20260924 16:00:39.934160 27614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 16:00:39.937361 27628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:39.938794 27614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:00:39.938974 27614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "e8631d30b519410ebead93cd4ffe5a0e"
format_stamp: "Formatted at 2026-09-24 16:00:39 on dist-test-slave-xn4h"
server_key: "27d5316ce0584b2a8a8ed328b7d3293e"
server_key_iv: "cedb037c9c6fa13414452b98a316dffd"
server_key_version: "encryptionkey@0"
I20260924 16:00:39.939193 27614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:39.944629 27614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:39.945416 27614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:39.945693 27614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:39.946658 27614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:39.947978 27614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:39.948104 27614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:39.948238 27614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:39.948326 27614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:39.981303 27614 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:46431
I20260924 16:00:39.981353 27742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:46431 every 8 connection(s)
I20260924 16:00:39.982735 27614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:00:39.983214 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 27614
I20260924 16:00:39.983410 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:00:39.984015 17318 external_mini_cluster.cc:1428] Setting key 0dff1b46ca726100a0a4f9029df90314
I20260924 16:00:39.991525 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:39.985731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:58853 (local address 127.16.233.190:37015)
0924 16:00:39.986008 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:39.986014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:39.987823 (+ 1809us) server_negotiation.cc:407] Connection header received
0924 16:00:39.989536 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:39.989540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:39.989590 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:39.989671 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:39.990833 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:39.990837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:39.990840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:39.990855 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:39.990885 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:39.990896 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:39.990984 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:39.991175 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 16:00:39.991226 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260924 16:00:39.992405 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:39.986515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:37015 (local address 127.16.233.131:58853)
0924 16:00:39.987651 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:39.987701 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:39.989303 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:39.989825 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:39.989832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:39.989920 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:39.990573 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:00:39.990576 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:39.990611 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:00:39.990685 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:39.990960 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:39.990966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:39.990970 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:00:39.991273 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 16:00:39.991489 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":372,"threads_started":1}
I20260924 16:00:39.993690 27743 heartbeater.cc:347] Connected to a master server at 127.16.233.190:37015
I20260924 16:00:39.993976 27743 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:39.994601 27743 heartbeater.cc:511] Master 127.16.233.190:37015 requested a full tablet report, sending...
I20260924 16:00:39.995946 27290 ts_manager.cc:194] Registered new tserver with Master: e8631d30b519410ebead93cd4ffe5a0e (127.16.233.131:46431)
I20260924 16:00:39.999482 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:00:40.005419 27480 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:00:40.004165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.16.233.190:37015)
0924 16:00:40.004348 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:40.004352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:40.004711 (+ 359us) server_negotiation.cc:407] Connection header received
0924 16:00:40.004834 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:40.004838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:40.004882 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:40.004964 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:40.005246 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39994: BlockingRecv error: recv got EOF from 127.0.0.1:39994 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:00:40.006212 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 27349
I20260924 16:00:40.009581 27752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:40.007084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.16.233.129:38193)
0924 16:00:40.007647 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:40.007653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:40.007677 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:00:40.007732 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:40.007738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:40.007915 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:40.008031 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:40.008973 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:40.008980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:40.008990 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:40.009007 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:40.009066 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:40.009081 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:40.009184 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:40.009273 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 16:00:40.009349 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":272,"threads_started":1}
I20260924 16:00:40.010758 27473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:40.093472 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 27349
I20260924 16:00:40.110937 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 27482
I20260924 16:00:40.114599 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:40.111850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.16.233.130:40891)
0924 16:00:40.112378 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:40.112384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:40.112431 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:00:40.112495 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:40.112516 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:40.112749 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:40.112882 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:40.113821 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:40.113827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:40.113839 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:40.113852 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:40.113897 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:40.113933 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:40.114039 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:40.114220 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 16:00:40.114371 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
I20260924 16:00:40.115868 27606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:40.201963 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 27482
I20260924 16:00:40.217733 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 27614
I20260924 16:00:40.221972 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:40.218858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.16.233.131:46431)
0924 16:00:40.219532 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:40.219538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:40.219568 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:00:40.219632 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:40.219639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:40.219847 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:40.220013 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:40.221170 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:40.221177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:40.221188 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:00:40.221200 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:40.221242 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:00:40.221255 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:40.221439 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:40.221552 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 16:00:40.221659 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":346,"threads_started":1}
I20260924 16:00:40.223256 27737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:40.304283 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 27614
I20260924 16:00:40.320287 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 27258
I20260924 16:00:40.321483 27320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:40.423852 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 27258
2026-09-24T16:00:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2346 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:00:40 dist-test-slave-xn4h krb5kdc[27761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:00:40 dist-test-slave-xn4h krb5kdc[27761](info): set up 2 sockets
Sep 24 16:00:40 dist-test-slave-xn4h krb5kdc[27761](info): commencing operation
krb5kdc: starting...
W20260924 16:00:41.536612 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:00:41 dist-test-slave-xn4h krb5kdc[27761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:00:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:00:41.755460 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36343
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:42.072919 27774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:42.073308 27774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:42.073480 27774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:42.080502 27774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:00:42.080637 27774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:42.080744 27774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:42.080853 27774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:00:42.080936 27774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:00:42.090248 27774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:42.091636 27774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:42.093331 27774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:42.105573 27780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:42.105609 27779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:42.105630 27782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:42.107604 27774 server_base.cc:1034] running on GCE node
I20260924 16:00:42.108368 27774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:42.109705 27774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:42.110936 27774 hybrid_clock.cc:648] HybridClock initialized: now 1790265642110894 us; error 33 us; skew 500 ppm
I20260924 16:00:42.111276 27774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:42 dist-test-slave-xn4h krb5kdc[27761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265642, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:42.114403 27774 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:00:42.115932 27774 webserver.cc:460] Webserver started at http://127.16.233.190:40241/ using document root <none> and password file <none>
I20260924 16:00:42.116715 27774 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:42.116874 27774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:42.117183 27774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:42.120147 27774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "e227a4fbdbcc407484737baf2ca6df36"
format_stamp: "Formatted at 2026-09-24 16:00:42 on dist-test-slave-xn4h"
server_key: "27474517343d1573ad7743a3595c1ab6"
server_key_iv: "7e538874220ae786fb67a98fed0c2470"
server_key_version: "encryptionkey@0"
I20260924 16:00:42.121115 27774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "e227a4fbdbcc407484737baf2ca6df36"
format_stamp: "Formatted at 2026-09-24 16:00:42 on dist-test-slave-xn4h"
server_key: "27474517343d1573ad7743a3595c1ab6"
server_key_iv: "7e538874220ae786fb67a98fed0c2470"
server_key_version: "encryptionkey@0"
I20260924 16:00:42.127501 27774 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 16:00:42.130731 27789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:42.131910 27774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:00:42.132090 27774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "e227a4fbdbcc407484737baf2ca6df36"
format_stamp: "Formatted at 2026-09-24 16:00:42 on dist-test-slave-xn4h"
server_key: "27474517343d1573ad7743a3595c1ab6"
server_key_iv: "7e538874220ae786fb67a98fed0c2470"
server_key_version: "encryptionkey@0"
I20260924 16:00:42.132277 27774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:42.165367 27774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:42.169883 27774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:42.170192 27774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:42.188510 27774 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:36343
I20260924 16:00:42.188546 27842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:36343 every 8 connection(s)
I20260924 16:00:42.189865 27774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:00:42.193521 27843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:42.196347 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 27774
I20260924 16:00:42.196589 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:00:42.197369 17318 external_mini_cluster.cc:1428] Setting key 0d6d6f3d1e173f59875d69897376309c
I20260924 16:00:42.205973 27843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36: Bootstrap starting.
I20260924 16:00:42.209589 27843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:42.210894 27843 log.cc:824] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:00:42 dist-test-slave-xn4h krb5kdc[27761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:00:42.214051 27843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36: No bootstrap required, opened a new log
I20260924 16:00:42.221364 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:42.199532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44434 (local address 127.16.233.190:36343)
0924 16:00:42.200349 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:42.200360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:42.200401 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:00:42.202015 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:42.202036 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:42.202489 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:42.202815 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:42.204545 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:42.205771 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:42.207623 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:42.208253 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:42.208394 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:42.213076 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:42.213091 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:42.213113 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:42.213155 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:42.217044 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:42.217668 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:42.217677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:42.217685 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:42.217894 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:42.218232 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:42.218237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:42.218241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:42.218959 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:42.219275 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:42.219561 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 16:00:42.220002 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":412,"threads_started":1}
I20260924 16:00:42.224509 27843 raft_consensus.cc:348] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e227a4fbdbcc407484737baf2ca6df36" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } }
I20260924 16:00:42.224826 27843 raft_consensus.cc:374] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:42.224898 27843 raft_consensus.cc:729] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e227a4fbdbcc407484737baf2ca6df36, State: Initialized, Role: FOLLOWER
I20260924 16:00:42.225412 27843 consensus_queue.cc:260] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e227a4fbdbcc407484737baf2ca6df36" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } }
I20260924 16:00:42.225610 27843 raft_consensus.cc:388] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:00:42.225706 27843 raft_consensus.cc:482] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:00:42.225812 27843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:42.227905 27843 raft_consensus.cc:504] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e227a4fbdbcc407484737baf2ca6df36" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } }
I20260924 16:00:42.228322 27843 leader_election.cc:302] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e227a4fbdbcc407484737baf2ca6df36; no voters:
I20260924 16:00:42.228864 27843 leader_election.cc:288] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:00:42.229022 27848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:42.230767 27848 raft_consensus.cc:686] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [term 1 LEADER]: Becoming Leader. State: Replica: e227a4fbdbcc407484737baf2ca6df36, State: Running, Role: LEADER
I20260924 16:00:42.231199 27843 sys_catalog.cc:564] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:00:42.231244 27848 consensus_queue.cc:237] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e227a4fbdbcc407484737baf2ca6df36" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } }
I20260924 16:00:42.234050 27851 sys_catalog.cc:455] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e227a4fbdbcc407484737baf2ca6df36. Latest consensus state: current_term: 1 leader_uuid: "e227a4fbdbcc407484737baf2ca6df36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e227a4fbdbcc407484737baf2ca6df36" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } } }
I20260924 16:00:42.234628 27851 sys_catalog.cc:458] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:42.234429 27849 sys_catalog.cc:455] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e227a4fbdbcc407484737baf2ca6df36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e227a4fbdbcc407484737baf2ca6df36" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36343 } } }
I20260924 16:00:42.234833 27849 sys_catalog.cc:458] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36 [sys.catalog]: This master's current role is: LEADER
I20260924 16:00:42.237614 27858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:00:42.242700 27858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:00:42.252049 27858 catalog_manager.cc:1347] Generated new cluster ID: b4b3694eed204353ad0213892fc982f5
I20260924 16:00:42.252172 27858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:00:42.267664 27858 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:00:42.269120 27858 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:00:42.280181 27858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36: Generated new TSK 0
I20260924 16:00:42.280936 27858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 16:00:42.343959 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36343
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:42.666507 27870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:42.666815 27870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:42.666996 27870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:42.674371 27870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:42.674496 27870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:42.674693 27870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:00:42.684147 27870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:42.685322 27870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:42.686631 27870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:42.699293 27876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:42.699325 27878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:42.699371 27875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:42.700199 27870 server_base.cc:1034] running on GCE node
I20260924 16:00:42.700860 27870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:42.701742 27870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:42.702961 27870 hybrid_clock.cc:648] HybridClock initialized: now 1790265642702917 us; error 53 us; skew 500 ppm
I20260924 16:00:42.703302 27870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:42 dist-test-slave-xn4h krb5kdc[27761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265642, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:42.706214 27870 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:00:42.707578 27870 webserver.cc:460] Webserver started at http://127.16.233.129:35175/ using document root <none> and password file <none>
I20260924 16:00:42.708223 27870 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:42.708381 27870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:42.708678 27870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:42.711524 27870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "023b3c17469840d3a7aeca0abd4e6372"
format_stamp: "Formatted at 2026-09-24 16:00:42 on dist-test-slave-xn4h"
server_key: "f9bb8ec4a3893d0a0744d51555c85886"
server_key_iv: "a6babcf5b5fcb0bf6dc8063354d8c352"
server_key_version: "encryptionkey@0"
I20260924 16:00:42.712296 27870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "023b3c17469840d3a7aeca0abd4e6372"
format_stamp: "Formatted at 2026-09-24 16:00:42 on dist-test-slave-xn4h"
server_key: "f9bb8ec4a3893d0a0744d51555c85886"
server_key_iv: "a6babcf5b5fcb0bf6dc8063354d8c352"
server_key_version: "encryptionkey@0"
I20260924 16:00:42.717223 27870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 16:00:42.720664 27885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:42.722009 27870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:00:42.722178 27870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "023b3c17469840d3a7aeca0abd4e6372"
format_stamp: "Formatted at 2026-09-24 16:00:42 on dist-test-slave-xn4h"
server_key: "f9bb8ec4a3893d0a0744d51555c85886"
server_key_iv: "a6babcf5b5fcb0bf6dc8063354d8c352"
server_key_version: "encryptionkey@0"
I20260924 16:00:42.722441 27870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:42.740043 27870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:42.743808 27870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:42.744094 27870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:42.744980 27870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:42.746292 27870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:42.746433 27870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:42.746603 27870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:42.746663 27870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:42.779753 27870 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:36503
I20260924 16:00:42.779863 28000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:36503 every 8 connection(s)
I20260924 16:00:42.781179 27870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:00:42.784790 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 27870
I20260924 16:00:42.785017 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:00:42.785760 17318 external_mini_cluster.cc:1428] Setting key d391a4ee89a317202d6eff3f7fe272ac
Sep 24 16:00:42 dist-test-slave-xn4h krb5kdc[27761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265642, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:00:42.801599 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:42.783756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:57489 (local address 127.16.233.190:36343)
0924 16:00:42.784153 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:42.784158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:42.785386 (+ 1228us) server_negotiation.cc:407] Connection header received
0924 16:00:42.787646 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:42.787651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:42.787711 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:42.787843 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:42.789281 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:42.790455 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:42.792027 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:42.792459 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:42.792600 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:42.796027 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:42.796033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:42.796036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:42.796069 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:42.798619 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:42.799327 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:42.799336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:42.799340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:42.799464 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:42.799882 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:42.799889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:42.799892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:42.800170 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:42.800326 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:42.801018 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 16:00:42.801236 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":191,"threads_started":1}
I20260924 16:00:42.802686 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:42.784338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.16.233.129:57489)
0924 16:00:42.785221 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:42.785275 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:42.787339 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:42.788054 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:42.788062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:42.788551 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:42.789091 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:42.789108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:42.790604 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:42.790608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:42.791887 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:42.791903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:42.792691 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:42.792700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:42.792952 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:42.793624 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:42.793650 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:42.795805 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:42.798760 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:42.798769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:42.798784 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:42.799193 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:42.799575 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:42.799580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:42.799584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:42.799771 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:42.800309 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:42.800317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:42.800648 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 16:00:42.800927 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 16:00:42.801462 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":320,"threads_started":1}
I20260924 16:00:42.804399 28001 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36343
I20260924 16:00:42.804708 28001 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:42.805480 28001 heartbeater.cc:511] Master 127.16.233.190:36343 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 16:00:42.807703 27806 ts_manager.cc:194] Registered new tserver with Master: 023b3c17469840d3a7aeca0abd4e6372 (127.16.233.129:36503)
I20260924 16:00:42.809401 27806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:57489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 16:00:42.837001 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36343
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:43.159624 28007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:43.159932 28007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:43.160132 28007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:43.167309 28007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:43.167465 28007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:43.167656 28007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:00:43.177302 28007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:43.178622 28007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:43.179962 28007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:43.193290 28013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:43.193962 28012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:43.194096 28015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:43.194710 28007 server_base.cc:1034] running on GCE node
I20260924 16:00:43.195343 28007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:43.196286 28007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:43.197520 28007 hybrid_clock.cc:648] HybridClock initialized: now 1790265643197465 us; error 50 us; skew 500 ppm
I20260924 16:00:43.197849 28007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:43 dist-test-slave-xn4h krb5kdc[27761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265643, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:43.201252 28007 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:00:43.202812 28007 webserver.cc:460] Webserver started at http://127.16.233.130:44589/ using document root <none> and password file <none>
I20260924 16:00:43.203485 28007 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:43.203629 28007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:43.203943 28007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:43.206820 28007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "c8ace3b1e8464fb2b7579d80cc0914b3"
format_stamp: "Formatted at 2026-09-24 16:00:43 on dist-test-slave-xn4h"
server_key: "ef34b51cdd0309851b4d7ba774c18767"
server_key_iv: "a514c0d6779905d89e7e99c8efa844c0"
server_key_version: "encryptionkey@0"
I20260924 16:00:43.207604 28007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "c8ace3b1e8464fb2b7579d80cc0914b3"
format_stamp: "Formatted at 2026-09-24 16:00:43 on dist-test-slave-xn4h"
server_key: "ef34b51cdd0309851b4d7ba774c18767"
server_key_iv: "a514c0d6779905d89e7e99c8efa844c0"
server_key_version: "encryptionkey@0"
I20260924 16:00:43.212576 28007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:00:43.216020 28022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:43.217337 28007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:00:43.217505 28007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "c8ace3b1e8464fb2b7579d80cc0914b3"
format_stamp: "Formatted at 2026-09-24 16:00:43 on dist-test-slave-xn4h"
server_key: "ef34b51cdd0309851b4d7ba774c18767"
server_key_iv: "a514c0d6779905d89e7e99c8efa844c0"
server_key_version: "encryptionkey@0"
I20260924 16:00:43.217701 28007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:43.234076 28007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:43.238128 28007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:43.238469 28007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:43.239385 28007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:43.240733 28007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:43.240837 28007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:43.240984 28007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:43.241055 28007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:43.273721 28007 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:43909
I20260924 16:00:43.273777 28136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:43909 every 8 connection(s)
I20260924 16:00:43.275043 28007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:00:43.278699 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 28007
I20260924 16:00:43.278908 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:00:43.279354 17318 external_mini_cluster.cc:1428] Setting key c51e9f36f72923af3167518d5eebad4d
Sep 24 16:00:43 dist-test-slave-xn4h krb5kdc[27761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265643, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:00:43.296180 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.277576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:60867 (local address 127.16.233.190:36343)
0924 16:00:43.277750 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:43.277754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:43.278816 (+ 1062us) server_negotiation.cc:407] Connection header received
0924 16:00:43.281051 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:43.281056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:43.281120 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:43.281219 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:43.282867 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.283554 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.285078 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.285480 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.285618 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.289071 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:43.289080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:43.289088 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:43.289134 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:43.292782 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:43.293604 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:43.293612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:43.293619 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:43.293793 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:43.294207 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:43.294215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:43.294221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:43.294614 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:43.294777 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:43.295444 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 16:00:43.295713 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:00:43.297171 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.277904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.16.233.130:60867)
0924 16:00:43.278661 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:43.278703 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:43.280760 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:43.281503 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:43.281514 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:43.282042 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:43.282664 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.282682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.283728 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.283732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.284932 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.284945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.285704 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.285710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.285941 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.286661 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:43.286683 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:43.288850 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:43.292923 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:43.292930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:43.292942 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:43.293464 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:43.293908 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:43.293911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:43.293914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:43.294087 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:43.294759 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:43.294772 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:43.295076 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 16:00:43.295296 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 16:00:43.295670 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":319,"threads_started":1}
I20260924 16:00:43.298802 28137 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36343
I20260924 16:00:43.299156 28137 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:43.300143 28137 heartbeater.cc:511] Master 127.16.233.190:36343 requested a full tablet report, sending...
I20260924 16:00:43.301887 27806 ts_manager.cc:194] Registered new tserver with Master: c8ace3b1e8464fb2b7579d80cc0914b3 (127.16.233.130:43909)
I20260924 16:00:43.302810 27806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:60867
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 16:00:43.333067 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36343
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:00:43.657783 28142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:00:43.658138 28142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:00:43.658344 28142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:00:43.665771 28142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:00:43.665917 28142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:00:43.666095 28142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:00:43.675882 28142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:00:43.677295 28142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:00:43.678709 28142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:00:43.691725 28147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:43.691778 28148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:00:43.691808 28150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:00:43.692579 28142 server_base.cc:1034] running on GCE node
I20260924 16:00:43.693117 28142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:00:43.694067 28142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:00:43.695300 28142 hybrid_clock.cc:648] HybridClock initialized: now 1790265643695228 us; error 71 us; skew 500 ppm
I20260924 16:00:43.695617 28142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:00:43 dist-test-slave-xn4h krb5kdc[27761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265643, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:00:43.698622 28142 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:00:43.699955 28142 webserver.cc:460] Webserver started at http://127.16.233.131:39399/ using document root <none> and password file <none>
I20260924 16:00:43.700613 28142 fs_manager.cc:359] Metadata directory not provided
I20260924 16:00:43.700759 28142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:00:43.701054 28142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:00:43.703966 28142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "0a3eb965d70f423ba43c946346b28b1b"
format_stamp: "Formatted at 2026-09-24 16:00:43 on dist-test-slave-xn4h"
server_key: "8b5cdf1c11bdf1b1c00e57a6075d73b4"
server_key_iv: "2ad3987f1a3148d631e294043733b38b"
server_key_version: "encryptionkey@0"
I20260924 16:00:43.704703 28142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "0a3eb965d70f423ba43c946346b28b1b"
format_stamp: "Formatted at 2026-09-24 16:00:43 on dist-test-slave-xn4h"
server_key: "8b5cdf1c11bdf1b1c00e57a6075d73b4"
server_key_iv: "2ad3987f1a3148d631e294043733b38b"
server_key_version: "encryptionkey@0"
I20260924 16:00:43.709560 28142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 16:00:43.712927 28157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:43.714216 28142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:00:43.714408 28142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "0a3eb965d70f423ba43c946346b28b1b"
format_stamp: "Formatted at 2026-09-24 16:00:43 on dist-test-slave-xn4h"
server_key: "8b5cdf1c11bdf1b1c00e57a6075d73b4"
server_key_iv: "2ad3987f1a3148d631e294043733b38b"
server_key_version: "encryptionkey@0"
I20260924 16:00:43.714619 28142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:00:43.730257 28142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:00:43.734241 28142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:00:43.734632 28142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:00:43.735728 28142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:00:43.737150 28142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:00:43.737246 28142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:43.737331 28142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:00:43.737376 28142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:00:43.770411 28142 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:43723
I20260924 16:00:43.770455 28269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:43723 every 8 connection(s)
I20260924 16:00:43.771682 28142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:00:43.775624 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 28142
I20260924 16:00:43.775841 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:00:43.776420 17318 external_mini_cluster.cc:1428] Setting key a176f5363b97db9bea247d8c2d77599e
Sep 24 16:00:43 dist-test-slave-xn4h krb5kdc[27761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265643, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:00:43.793254 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.774203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:41637 (local address 127.16.233.190:36343)
0924 16:00:43.774407 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:43.774412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:43.775724 (+ 1312us) server_negotiation.cc:407] Connection header received
0924 16:00:43.778224 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:43.778231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:43.778349 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:43.778455 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:43.780464 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.781200 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.783450 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.783871 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.783968 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.787459 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:43.787464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:43.787468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:43.787500 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:43.790095 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:43.791045 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:43.791049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:43.791051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:43.791238 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:43.791732 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:43.791737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:43.791741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:43.792102 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:43.792223 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:43.792849 (+ 626us) server_negotiation.cc:299] Negotiation successful
0924 16:00:43.793011 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:00:43.794174 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.774701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36343 (local address 127.16.233.131:41637)
0924 16:00:43.775526 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:43.775575 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:43.777885 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:43.778739 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:43.778748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:43.779467 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:00:43.780269 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.780294 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.781358 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.781364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.783309 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.783325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.784102 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.784112 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.784391 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.785076 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:00:43.785110 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:00:43.787271 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:00:43.790290 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:43.790299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:43.790331 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:43.790921 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:43.791369 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:00:43.791374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:00:43.791378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:00:43.791604 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:00:43.792224 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:00:43.792232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:00:43.792531 (+ 299us) client_negotiation.cc:769] Sending connection context
0924 16:00:43.792753 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 16:00:43.793116 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":273,"threads_started":1}
I20260924 16:00:43.795449 28270 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36343
I20260924 16:00:43.795756 28270 heartbeater.cc:464] Registering TS with master...
I20260924 16:00:43.796466 28270 heartbeater.cc:511] Master 127.16.233.190:36343 requested a full tablet report, sending...
I20260924 16:00:43.797811 27806 ts_manager.cc:194] Registered new tserver with Master: 0a3eb965d70f423ba43c946346b28b1b (127.16.233.131:43723)
I20260924 16:00:43.798667 27806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:41637
I20260924 16:00:43.804003 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:00:43.813163 28001 heartbeater.cc:503] Master 127.16.233.190:36343 was elected leader, sending a full tablet report...
I20260924 16:00:43.820585 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.808478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.16.233.190:36343)
0924 16:00:43.808664 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:43.808668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:43.809135 (+ 467us) server_negotiation.cc:407] Connection header received
0924 16:00:43.809281 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:43.809286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:43.809345 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:43.809426 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:43.810867 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.811589 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.812738 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.813149 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.813303 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.815859 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:43.815864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:43.815867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:43.815900 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:43.818254 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:43.818923 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:43.818927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:43.818929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:43.819055 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:43.819426 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:43.819429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:43.819431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:43.819739 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:43.819920 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:43.820196 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:00:43.820363 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:00:43.824673 27806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:00:43.831519 27806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:00:43.869451 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.844553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:43723 (local address 127.0.0.1:56478)
0924 16:00:43.860760 (+ 16207us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:43.860785 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:43.860933 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:43.862173 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:43.862176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:43.862196 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:43.862690 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.862709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.864680 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.864684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.867115 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.867126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.868405 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.868412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.868592 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.868650 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 16:00:43.868799 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 16:00:43.868991 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16021,"mutex_wait_us":29,"thread_start_us":16631,"threads_started":1}
I20260924 16:00:43.869920 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.843910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:43909 (local address 127.0.0.1:43216)
0924 16:00:43.844713 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:43.844754 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:43.844936 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:43.861511 (+ 16575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:43.861520 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:43.861556 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:43.862052 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.862066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.864257 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.864271 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.866929 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.866945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.868764 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.868773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.868966 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.868996 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:00:43.869085 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:00:43.869219 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":89,"thread_start_us":3291,"threads_started":1}
I20260924 16:00:43.869979 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.844088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.16.233.130:43909)
0924 16:00:43.859826 (+ 15738us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:43.859839 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:43.859877 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:00:43.860008 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:43.860018 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:43.860365 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:43.862282 (+ 1917us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:43.862548 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.864118 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.867101 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.868526 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.868720 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.868884 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:43.869154 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 16:00:43.869322 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15557,"thread_start_us":15409,"threads_started":1}
I20260924 16:00:43.870224 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.845379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.16.233.131:43723)
0924 16:00:43.861214 (+ 15835us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:43.861222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:43.861257 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:00:43.861362 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:43.861370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:43.861649 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:43.861874 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:43.862953 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.864542 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.867280 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.868177 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.868409 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.868596 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:43.869732 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0924 16:00:43.869920 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15641,"thread_start_us":16460,"threads_started":1}
I20260924 16:00:43.870445 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.845473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.16.233.129:36503)
0924 16:00:43.860277 (+ 14804us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:43.860284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:43.861778 (+ 1494us) server_negotiation.cc:407] Connection header received
0924 16:00:43.861881 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:43.861889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:43.862129 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:43.862375 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:43.863059 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.864149 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.866864 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.868351 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.868553 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.868739 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:43.869666 (+ 927us) server_negotiation.cc:299] Negotiation successful
0924 16:00:43.869856 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14630,"thread_start_us":14457,"threads_started":1}
I20260924 16:00:43.870831 28284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:43.844601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.0.0.1:34494)
0924 16:00:43.861508 (+ 16907us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:43.861524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:43.861646 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:43.862387 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:43.862391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:43.862415 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:43.862881 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.862893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.864313 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.864318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.866689 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:43.866705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:43.868900 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:43.868909 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:43.869039 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:43.869094 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 16:00:43.869188 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:00:43.869328 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16758,"mutex_wait_us":77,"thread_start_us":15816,"threads_started":1}
I20260924 16:00:43.873924 27934 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a7491947d48708500698aed48b066 (DEFAULT_TABLE table=test-table [id=03c4ed8a8151494fbca113fc5772907d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:43.873967 28205 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a7491947d48708500698aed48b066 (DEFAULT_TABLE table=test-table [id=03c4ed8a8151494fbca113fc5772907d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:43.873967 28070 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a7491947d48708500698aed48b066 (DEFAULT_TABLE table=test-table [id=03c4ed8a8151494fbca113fc5772907d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:00:43.875397 28205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a7491947d48708500698aed48b066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:43.875397 28070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a7491947d48708500698aed48b066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:43.875470 27934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a7491947d48708500698aed48b066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:00:43.886854 28288 tablet_bootstrap.cc:492] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372: Bootstrap starting.
I20260924 16:00:43.890029 28291 tablet_bootstrap.cc:492] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: Bootstrap starting.
I20260924 16:00:43.891124 28288 tablet_bootstrap.cc:654] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:43.891449 28289 tablet_bootstrap.cc:492] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: Bootstrap starting.
I20260924 16:00:43.892947 28288 log.cc:824] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:43.894587 28291 tablet_bootstrap.cc:654] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:43.894949 28289 tablet_bootstrap.cc:654] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: Neither blocks nor log segments found. Creating new log.
I20260924 16:00:43.895689 28291 log.cc:824] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:43.896098 28289 log.cc:824] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:00:43.896329 28288 tablet_bootstrap.cc:492] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372: No bootstrap required, opened a new log
I20260924 16:00:43.896628 28288 ts_tablet_manager.cc:1397] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 16:00:43.898203 28291 tablet_bootstrap.cc:492] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: No bootstrap required, opened a new log
I20260924 16:00:43.898492 28291 ts_tablet_manager.cc:1397] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:00:43.904942 28289 tablet_bootstrap.cc:492] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: No bootstrap required, opened a new log
I20260924 16:00:43.905258 28289 ts_tablet_manager.cc:1397] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260924 16:00:43.907086 28288 raft_consensus.cc:348] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } }
I20260924 16:00:43.907430 28288 raft_consensus.cc:374] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:43.907523 28288 raft_consensus.cc:729] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023b3c17469840d3a7aeca0abd4e6372, State: Initialized, Role: FOLLOWER
I20260924 16:00:43.907496 28291 raft_consensus.cc:348] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } }
I20260924 16:00:43.907886 28291 raft_consensus.cc:374] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:43.907966 28291 raft_consensus.cc:729] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3eb965d70f423ba43c946346b28b1b, State: Initialized, Role: FOLLOWER
I20260924 16:00:43.908134 28288 consensus_queue.cc:260] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } }
I20260924 16:00:43.908478 28291 consensus_queue.cc:260] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } }
I20260924 16:00:43.909976 28288 ts_tablet_manager.cc:1428] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260924 16:00:43.910851 28270 heartbeater.cc:503] Master 127.16.233.190:36343 was elected leader, sending a full tablet report...
I20260924 16:00:43.911437 28291 ts_tablet_manager.cc:1428] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: Time spent starting tablet: real 0.013s user 0.002s sys 0.010s
I20260924 16:00:43.915395 28289 raft_consensus.cc:348] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } }
I20260924 16:00:43.915822 28289 raft_consensus.cc:374] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:00:43.915913 28289 raft_consensus.cc:729] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ace3b1e8464fb2b7579d80cc0914b3, State: Initialized, Role: FOLLOWER
I20260924 16:00:43.916383 28289 consensus_queue.cc:260] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } }
I20260924 16:00:43.917241 28137 heartbeater.cc:503] Master 127.16.233.190:36343 was elected leader, sending a full tablet report...
I20260924 16:00:43.917748 28289 ts_tablet_manager.cc:1428] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260924 16:00:44.023317 28271 tablet.cc:2369] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:00:44.026944 28138 tablet.cc:2369] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:00:44.033600 28002 tablet.cc:2369] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:00:44.133657 28296 raft_consensus.cc:482] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:00:44.133929 28296 raft_consensus.cc:504] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } }
I20260924 16:00:44.135813 28296 leader_election.cc:288] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8ace3b1e8464fb2b7579d80cc0914b3 (127.16.233.130:43909), 023b3c17469840d3a7aeca0abd4e6372 (127.16.233.129:36503)
I20260924 16:00:44.143272 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.136544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:52705 (local address 127.16.233.130:43909)
0924 16:00:44.136749 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:44.136753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:44.136769 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:00:44.136896 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:44.136901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:44.136980 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:44.137075 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:44.137638 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.138410 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.141474 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.142205 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.142371 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:44.142486 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:44.142872 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 16:00:44.143016 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:00:44.143283 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.136060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:43909 (local address 127.16.233.131:52705)
0924 16:00:44.136286 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.136307 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:44.136446 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:44.137141 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:44.137149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:44.137193 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:44.137475 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:44.137488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.138561 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.138569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:44.141307 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:44.141328 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.142380 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.142393 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:44.142678 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:44.142722 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 16:00:44.142847 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 16:00:44.142996 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 16:00:44.144425 28090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a7491947d48708500698aed48b066" candidate_uuid: "0a3eb965d70f423ba43c946346b28b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" is_pre_election: true
I20260924 16:00:44.144999 28090 raft_consensus.cc:2454] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a3eb965d70f423ba43c946346b28b1b in term 0.
I20260924 16:00:44.145771 28158 leader_election.cc:302] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3eb965d70f423ba43c946346b28b1b, c8ace3b1e8464fb2b7579d80cc0914b3; no voters:
I20260924 16:00:44.146162 28296 raft_consensus.cc:2781] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:00:44.146314 28296 raft_consensus.cc:482] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:00:44.146440 28296 raft_consensus.cc:3037] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:44.149063 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.136253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.16.233.131:51761)
0924 16:00:44.142838 (+ 6585us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.142856 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:00:44.143052 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:00:44.143376 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:00:44.143381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:00:44.143405 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:00:44.143759 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:44.143770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.145105 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.145109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:44.146858 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:00:44.146871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.148460 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.148464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:00:44.148589 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:44.148614 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:00:44.148694 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 16:00:44.148784 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6448,"thread_start_us":6236,"threads_started":1}
I20260924 16:00:44.149132 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.136797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:51761 (local address 127.16.233.129:36503)
0924 16:00:44.137000 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:44.137008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:44.142950 (+ 5942us) server_negotiation.cc:407] Connection header received
0924 16:00:44.143180 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:44.143196 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:44.143265 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:44.143363 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:00:44.143951 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.144964 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.147034 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.148255 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.148436 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:44.148510 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:44.148754 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 16:00:44.148891 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:00:44.149857 28296 raft_consensus.cc:504] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } }
I20260924 16:00:44.150028 27954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a7491947d48708500698aed48b066" candidate_uuid: "0a3eb965d70f423ba43c946346b28b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023b3c17469840d3a7aeca0abd4e6372" is_pre_election: true
I20260924 16:00:44.150610 27954 raft_consensus.cc:2454] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a3eb965d70f423ba43c946346b28b1b in term 0.
I20260924 16:00:44.150740 28296 leader_election.cc:288] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [CANDIDATE]: Term 1 election: Requested vote from peers c8ace3b1e8464fb2b7579d80cc0914b3 (127.16.233.130:43909), 023b3c17469840d3a7aeca0abd4e6372 (127.16.233.129:36503)
I20260924 16:00:44.150926 28090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a7491947d48708500698aed48b066" candidate_uuid: "0a3eb965d70f423ba43c946346b28b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3"
I20260924 16:00:44.151093 27953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a7491947d48708500698aed48b066" candidate_uuid: "0a3eb965d70f423ba43c946346b28b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023b3c17469840d3a7aeca0abd4e6372"
I20260924 16:00:44.151168 28090 raft_consensus.cc:3037] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:44.151325 27953 raft_consensus.cc:3037] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:00:44.153396 27953 raft_consensus.cc:2454] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a3eb965d70f423ba43c946346b28b1b in term 1.
I20260924 16:00:44.153393 28090 raft_consensus.cc:2454] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a3eb965d70f423ba43c946346b28b1b in term 1.
I20260924 16:00:44.153777 28158 leader_election.cc:302] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3eb965d70f423ba43c946346b28b1b, c8ace3b1e8464fb2b7579d80cc0914b3; no voters:
I20260924 16:00:44.154096 28296 raft_consensus.cc:2781] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:00:44.154575 28296 raft_consensus.cc:686] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 1 LEADER]: Becoming Leader. State: Replica: 0a3eb965d70f423ba43c946346b28b1b, State: Running, Role: LEADER
I20260924 16:00:44.155102 28296 consensus_queue.cc:237] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } }
I20260924 16:00:44.159628 27806 catalog_manager.cc:5495] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a3eb965d70f423ba43c946346b28b1b (127.16.233.131). New cstate: current_term: 1 leader_uuid: "0a3eb965d70f423ba43c946346b28b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a3eb965d70f423ba43c946346b28b1b" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 43723 } health_report { overall_health: HEALTHY } } }
I20260924 16:00:44.227610 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.222749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.16.233.131:43723)
0924 16:00:44.222924 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:44.222929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:44.223071 (+ 142us) server_negotiation.cc:407] Connection header received
0924 16:00:44.223224 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:44.223228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:44.223308 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:44.223432 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:00:44.223710 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.224421 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.225677 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.226108 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.226201 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:44.226701 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:00:44.227184 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:00:44.227293 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:44.227363 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 16:00:44.227428 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:00:44.235459 27953 raft_consensus.cc:1261] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [term 1 FOLLOWER]: Refusing update from remote peer 0a3eb965d70f423ba43c946346b28b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:44.235667 28090 raft_consensus.cc:1261] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [term 1 FOLLOWER]: Refusing update from remote peer 0a3eb965d70f423ba43c946346b28b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:00:44.236629 28299 consensus_queue.cc:1035] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ace3b1e8464fb2b7579d80cc0914b3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:44.236948 28296 consensus_queue.cc:1035] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "023b3c17469840d3a7aeca0abd4e6372" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:00:44.245484 28302 mvcc.cc:205] Tried to move back new op lower bound from 7332928078778961920 to 7332928078463610880. Current Snapshot: MvccSnapshot[applied={T|T < 7332928078778961920}]
I20260924 16:00:44.250720 28303 mvcc.cc:205] Tried to move back new op lower bound from 7332928078778961920 to 7332928078463610880. Current Snapshot: MvccSnapshot[applied={T|T < 7332928078778961920}]
I20260924 16:00:44.267973 27806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:44.268188 27806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:00:44.271564 27806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e227a4fbdbcc407484737baf2ca6df36: Sending DeleteTablet for 3 replicas of tablet 8e6a7491947d48708500698aed48b066
I20260924 16:00:44.272709 27934 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6a7491947d48708500698aed48b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:44 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:34494
I20260924 16:00:44.273000 28205 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6a7491947d48708500698aed48b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:44 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:56478
I20260924 16:00:44.273412 28070 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6a7491947d48708500698aed48b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:00:44 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:43216
I20260924 16:00:44.274993 28312 tablet_replica.cc:330] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: stopping tablet replica
I20260924 16:00:44.275444 28312 raft_consensus.cc:2229] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:00:44.274272 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 27870
I20260924 16:00:44.276372 28312 raft_consensus.cc:2258] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:44.279702 28312 ts_tablet_manager.cc:1905] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:44.287228 28312 ts_tablet_manager.cc:1918] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:44.287384 28312 log.cc:1196] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/8e6a7491947d48708500698aed48b066
Sep 24 16:00:44 dist-test-slave-xn4h krb5kdc[27761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 16:00:44.287919 28312 ts_tablet_manager.cc:1939] T 8e6a7491947d48708500698aed48b066 P 0a3eb965d70f423ba43c946346b28b1b: Deleting consensus metadata
I20260924 16:00:44.289688 27793 catalog_manager.cc:4841] TS 0a3eb965d70f423ba43c946346b28b1b (127.16.233.131:43723): tablet 8e6a7491947d48708500698aed48b066 (table test-table [id=03c4ed8a8151494fbca113fc5772907d]) successfully deleted
I20260924 16:00:44.290800 28313 tablet_replica.cc:330] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: stopping tablet replica
I20260924 16:00:44.291260 28313 raft_consensus.cc:2229] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:44.291849 28313 raft_consensus.cc:2258] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:44.292309 28311 tablet_replica.cc:330] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372: stopping tablet replica
I20260924 16:00:44.292687 28311 raft_consensus.cc:2229] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:00:44.293144 28311 raft_consensus.cc:2258] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:00:44.295195 28311 ts_tablet_manager.cc:1905] T 8e6a7491947d48708500698aed48b066 P 023b3c17469840d3a7aeca0abd4e6372: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:44.297286 28313 ts_tablet_manager.cc:1905] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:00:44.299122 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.277251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.16.233.129:36503)
0924 16:00:44.277437 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:44.277441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:44.277455 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:00:44.277497 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:44.277500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:44.277553 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:44.277660 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:44.279641 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.280777 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.282732 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.283339 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.283442 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:44.289000 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:44.289015 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:44.289031 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:44.289074 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:44.293028 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:44.294048 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:44.294059 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:44.294066 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:44.294248 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:44.294743 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:44.294748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:44.294751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:44.295169 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:44.296043 (+ 874us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:44.298529 (+ 2486us) server_negotiation.cc:299] Negotiation successful
0924 16:00:44.298744 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:00:44.300827 27995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:44.311851 28313 ts_tablet_manager.cc:1918] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:00:44.312177 28313 log.cc:1196] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/8e6a7491947d48708500698aed48b066
I20260924 16:00:44.312731 28313 ts_tablet_manager.cc:1939] T 8e6a7491947d48708500698aed48b066 P c8ace3b1e8464fb2b7579d80cc0914b3: Deleting consensus metadata
I20260924 16:00:44.314826 27790 catalog_manager.cc:4841] TS c8ace3b1e8464fb2b7579d80cc0914b3 (127.16.233.130:43909): tablet 8e6a7491947d48708500698aed48b066 (table test-table [id=03c4ed8a8151494fbca113fc5772907d]) successfully deleted
I20260924 16:00:44.422293 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 27870
W20260924 16:00:44.440483 27792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.129:36503 (error 108)
I20260924 16:00:44.440752 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 28007
I20260924 16:00:44.443899 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.443170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.0.0.1:34510)
0924 16:00:44.443621 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.443739 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":249,"threads_started":1}
W20260924 16:00:44.444212 27792 catalog_manager.cc:4584] TS 023b3c17469840d3a7aeca0abd4e6372 (127.16.233.129:36503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e6a7491947d48708500698aed48b066: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36503: connect: Connection refused (error 111)
Sep 24 16:00:44 dist-test-slave-xn4h krb5kdc[27761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:00:44.459071 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.442492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.16.233.130:43909)
0924 16:00:44.442670 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:44.442676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:44.442695 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:00:44.442817 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:44.442823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:44.442917 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:44.443069 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:44.444738 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.445429 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.447198 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.447589 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.447685 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:44.452514 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:44.452533 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:44.452553 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:44.452604 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:44.456274 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:44.456933 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:44.456943 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:44.456947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:44.457105 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:44.457468 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:44.457474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:44.457478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:44.457957 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:44.458252 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:44.458470 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 16:00:44.458695 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:00:44.459847 28132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:44.497889 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.497333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.0.0.1:34526)
0924 16:00:44.497572 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.497682 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 16:00:44.499020 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.498628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.0.0.1:34532)
0924 16:00:44.498785 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.498867 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 16:00:44.549405 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 28007
I20260924 16:00:44.559950 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.559359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.0.0.1:34546)
0924 16:00:44.559623 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.559751 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 16:00:44.561244 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.560791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.0.0.1:34554)
0924 16:00:44.560977 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.561072 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 16:00:44.568137 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 28142
Sep 24 16:00:44 dist-test-slave-xn4h krb5kdc[27761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 16:00:44.583839 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.568973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.16.233.131:43723)
0924 16:00:44.569184 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:00:44.569191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:00:44.569212 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:00:44.569373 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:00:44.569383 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:00:44.569457 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:00:44.569574 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:00:44.570929 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.571751 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.572995 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:00:44.573448 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:00:44.573611 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:00:44.577904 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:00:44.577917 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:00:44.577932 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:00:44.577970 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:00:44.580975 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:44.581637 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:44.581644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:44.581646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:44.581816 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:00:44.582198 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:00:44.582211 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:00:44.582213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:00:44.582633 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:00:44.583008 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0924 16:00:44.583234 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:00:44.583477 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:00:44.584519 28265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:44.642060 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.641576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.0.0.1:34568)
0924 16:00:44.641809 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.641899 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 16:00:44.643195 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.642796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.0.0.1:34570)
0924 16:00:44.642989 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.643064 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 16:00:44.704476 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 28142
I20260924 16:00:44.724102 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 27774
I20260924 16:00:44.725281 27838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:00:44.835216 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:00:44.834626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36503 (local address 127.0.0.1:34574)
0924 16:00:44.834883 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 16:00:44.835019 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 16:00:44.835603 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 27774
2026-09-24T16:00:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4413 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:00:44 dist-test-slave-xn4h krb5kdc[28323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:00:44 dist-test-slave-xn4h krb5kdc[28323](info): set up 2 sockets
Sep 24 16:00:44 dist-test-slave-xn4h krb5kdc[28323](info): commencing operation
krb5kdc: starting...
W20260924 16:00:45.948236 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:00:46 dist-test-slave-xn4h krb5kdc[28323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:00:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/rangeradmin_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/rangeradmin_127.16.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/rangerlookup_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/rangerlookup_127.16.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/HTTP_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/HTTP_127.16.233.148@KRBTEST.COM.keytab.
I20260924 16:00:46.267503 17318 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRO1Jyr/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/postgres -l logfile start
2026-09-24 16:00:48.784 UTC [28354] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:00:48.784 UTC [28354] LOG: listening on IPv4 address "127.16.233.148", port 59823
2026-09-24 16:00:48.788 UTC [28354] LOG: listening on Unix socket "/tmp/.s.PGSQL.59823"
2026-09-24 16:00:48.808 UTC [28357] LOG: database system was shut down at 2026-09-24 16:00:47 UTC
2026-09-24 16:00:48.812 UTC [28354] LOG: database system is ready to accept connections
I20260924 16:00:49.772018 17318 mini_postgres.cc:96] Postgres bound to 59823
2026-09-24 16:00:49.791 UTC [28365] FATAL: database "slave" does not exist
127.16.233.148:59823 - accepting connections
I20260924 16:00:49.792686 17318 mini_ranger.cc:161] Starting Ranger...
I20260924 16:00:49.820391 17318 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:00:50.286110 17318 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:00:50.286214 17318 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-admin
2026-09-24 16:00:50,731 [I] DB FLAVOR :POSTGRES
2026-09-24 16:00:50,731 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:00:50,732 [I] Checking connection..
2026-09-24 16:00:50,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:00:51,013 [I] Checking connection passed.
2026-09-24 16:00:51,013 [I] --------- Verifying version history table ---------
2026-09-24 16:00:51,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:00:51,272 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:00:51,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:00:51,499 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:00:51,499 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:00:51,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:00:51.743 UTC [28431] WARNING: there is no transaction in progress
2026-09-24 16:00:51,749 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:00:51,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:00:52,003 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:00:52,003 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:00:52,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:00:52,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:00:52,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xn4h.c.gcp-upstream.internal','N') ;"
2026-09-24 16:00:52,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:00:53,008 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:00:53,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:00:53,241 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:00:53,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:00:53,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:00:53,760 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:00:53,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:00:54.728 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.744 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.760 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.774 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.790 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.808 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.861 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.867 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.881 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.892 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.903 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.917 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.927 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.933 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.940 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:54.947 UTC [28539] WARNING: there is no transaction in progress
2026-09-24 16:00:55,574 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:00:55,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:00:55,805 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:00:55,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:00:56,070 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:00:56,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:00:56,328 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:00:56,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:00:56,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:00:56,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xn4h.c.gcp-upstream.internal';"
2026-09-24 16:00:57,094 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:00:57,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:00:57,345 [I] DB_PATCHES have already been applied
I20260924 16:00:57.352417 17318 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRO1Jyr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:00:57.352507 17318 mini_ranger.cc:193] Using host: 127.16.233.148
I20260924 16:00:57.367601 17318 mini_ranger.cc:227] Ranger admin URL: http://127.16.233.148:39735
Sep 24, 2026 4:00:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:00:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:00:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:00:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:00:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:00:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:00:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:00:58.490340 17318 mini_ranger.cc:160] Time spent starting Ranger: real 8.698s user 0.007s sys 0.034s
Sep 24, 2026 4:01:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:01:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:01:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:01:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:01:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:01:22.443625 17318 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/rangerkms_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/rangerkms_127.16.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.233.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.233.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.233.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/HTTP_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.233.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/HTTP_127.16.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:01:22 dist-test-slave-xn4h krb5kdc[28323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265682, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:01:22.616885 17318 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:01:22.639298 17318 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:01:23.496618 17318 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:01:23.496755 17318 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-kms
I20260924 16:01:23.496898 17318 mini_ranger_kms.cc:227] Using postgres at 127.16.233.148:59823
2026-09-24 16:01:24,336 [I] DB FLAVOR :POSTGRES
2026-09-24 16:01:24,336 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:01:24,336 [I] Checking connection
2026-09-24 16:01:24,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:01:24,630 [I] connection success
2026-09-24 16:01:24,631 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:01:24,631 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:01:24,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:01:24,871 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:01:24,871 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:01:24,871 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:01:24,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:59823/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:01:25,138 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:01:25.644213 17318 mini_ranger_kms.cc:310] Created kms service
I20260924 16:01:25.850695 17318 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:01:25.897403 17318 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:01:25 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265682, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:01:26.339887 17318 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:01:26.340255 17318 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:01:26.340327 17318 mini_ranger_kms.cc:240] Using host: 127.16.233.148
I20260924 16:01:26.358644 17318 mini_ranger_kms.cc:276] Ranger KMS PID: 28724
I20260924 16:01:26.358855 17318 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.233.148:40727
Sep 24, 2026 4:01:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:01:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:01:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:01:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:01:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:01:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 16:01:28.686766 17318 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.070s user 0.145s sys 0.378s
I20260924 16:01:28.687076 17318 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:01:28.687305 17318 mini_ranger_kms.cc:401] 127.16.233.148:40727/kms/v1/keys
Sep 24 16:01:28 dist-test-slave-xn4h krb5kdc[28323](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265688, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 16:01:29 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265688, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.148@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
Sep 24 16:01:29 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265688, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.148@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
Sep 24 16:01:32 dist-test-slave-xn4h krb5kdc[28323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:01:32.146091 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:35743
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:35743
--ranger_config_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:40727/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:32.560941 28775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:32.561266 28775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:32.561437 28775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:32.569087 28775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:01:32.569206 28775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:32.569290 28775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:32.569387 28775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:01:32.569471 28775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:01:32.582401 28775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.233.148:40727/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:35743
--ranger_config_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:35743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:01:32.583907 28775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:32.585280 28775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:32.598129 28780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:32.598068 28781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:32.598152 28783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:32.598774 28775 server_base.cc:1034] running on GCE node
I20260924 16:01:32.599583 28775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:32.601068 28775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:32.602345 28775 hybrid_clock.cc:648] HybridClock initialized: now 1790265692602257 us; error 67 us; skew 500 ppm
I20260924 16:01:32.602721 28775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:01:32 dist-test-slave-xn4h krb5kdc[28323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265692, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:32.612814 28775 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:01:32.614543 28775 webserver.cc:460] Webserver started at http://127.16.233.190:37853/ using document root <none> and password file <none>
I20260924 16:01:32.615198 28775 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:32.615298 28775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:32.615556 28775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:01:32 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265692, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:01:32.669451 28775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "1b2ece20d6084911a2f1aa041fbc1cf5"
format_stamp: "Formatted at 2026-09-24 16:01:32 on dist-test-slave-xn4h"
server_key: "8c94193aebb8dfce3d1ffb05d87b6d37"
server_key_iv: "b78082ffa7a89989497d0bd74e143cf8"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:32.670377 28775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "1b2ece20d6084911a2f1aa041fbc1cf5"
format_stamp: "Formatted at 2026-09-24 16:01:32 on dist-test-slave-xn4h"
server_key: "8c94193aebb8dfce3d1ffb05d87b6d37"
server_key_iv: "b78082ffa7a89989497d0bd74e143cf8"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:32.675830 28775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:01:32.692739 28790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:32.694442 28775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:01:32.694621 28775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "1b2ece20d6084911a2f1aa041fbc1cf5"
format_stamp: "Formatted at 2026-09-24 16:01:32 on dist-test-slave-xn4h"
server_key: "8c94193aebb8dfce3d1ffb05d87b6d37"
server_key_iv: "b78082ffa7a89989497d0bd74e143cf8"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:32.694878 28775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:32.728844 28775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:32.733911 28775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:32.734256 28775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:32.760092 28775 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:35743
I20260924 16:01:32.760345 28842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:35743 every 8 connection(s)
I20260924 16:01:32.761591 28775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:01:32.765575 28843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:32.767263 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 28775
I20260924 16:01:32.767485 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:01:32.768507 17318 external_mini_cluster.cc:1428] Setting key a6be3310c192f5e41735d12ff251471d
I20260924 16:01:32.779009 28843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5: Bootstrap starting.
I20260924 16:01:32.783106 28843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5: Neither blocks nor log segments found. Creating new log.
Sep 24 16:01:32 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:01:32.784262 28843 log.cc:824] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:32.787297 28843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5: No bootstrap required, opened a new log
I20260924 16:01:32.792510 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:32.771068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.16.233.190:35743)
0924 16:01:32.772196 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:32.772213 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:32.772261 (+ 48us) server_negotiation.cc:407] Connection header received
0924 16:01:32.773939 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:32.773962 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:32.774431 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:32.774768 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:32.776394 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:32.777751 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:32.779476 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:32.780163 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:32.780305 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:32.784875 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:32.784884 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:32.784899 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:32.784946 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:32.788120 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:32.788869 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:32.788875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:32.788884 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:32.789059 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:32.789559 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:32.789563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:32.789565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:32.790116 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:32.790493 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:32.790988 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 16:01:32.791416 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":600,"threads_started":1}
I20260924 16:01:32.798226 28843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2ece20d6084911a2f1aa041fbc1cf5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35743 } }
I20260924 16:01:32.798732 28843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:32.798859 28843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2ece20d6084911a2f1aa041fbc1cf5, State: Initialized, Role: FOLLOWER
I20260924 16:01:32.799438 28843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2ece20d6084911a2f1aa041fbc1cf5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35743 } }
I20260924 16:01:32.799678 28843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:01:32.799836 28843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:01:32.800006 28843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:32.802554 28843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2ece20d6084911a2f1aa041fbc1cf5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35743 } }
I20260924 16:01:32.803028 28843 leader_election.cc:302] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b2ece20d6084911a2f1aa041fbc1cf5; no voters:
I20260924 16:01:32.803601 28843 leader_election.cc:288] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:01:32.803759 28849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:32.805186 28849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [term 1 LEADER]: Becoming Leader. State: Replica: 1b2ece20d6084911a2f1aa041fbc1cf5, State: Running, Role: LEADER
I20260924 16:01:32.805874 28849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2ece20d6084911a2f1aa041fbc1cf5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35743 } }
I20260924 16:01:32.806088 28843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:01:32.809022 28850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b2ece20d6084911a2f1aa041fbc1cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2ece20d6084911a2f1aa041fbc1cf5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35743 } } }
I20260924 16:01:32.809020 28851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b2ece20d6084911a2f1aa041fbc1cf5. Latest consensus state: current_term: 1 leader_uuid: "1b2ece20d6084911a2f1aa041fbc1cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2ece20d6084911a2f1aa041fbc1cf5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35743 } } }
I20260924 16:01:32.809682 28851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:01:32.809682 28850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:01:32.810909 28857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:01:32.819018 28857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:01:32.831303 28843 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:01:32.832515 28857 catalog_manager.cc:1347] Generated new cluster ID: 24b0630c1449426bacabbb75f6591f37
I20260924 16:01:32.832623 28857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:01:32.856151 28857 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:01:32.857767 28857 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:01:32.874986 28857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5: Generated new TSK 0
I20260924 16:01:32.876009 28857 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 16:01:33.689978 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:40727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35743
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:01:33 dist-test-slave-xn4h krb5kdc[28323](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265693, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 16:01:34.189116 28889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:34.189626 28889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:34.189832 28889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:34.202085 28889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:34.202258 28889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:34.202538 28889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:01:34.219424 28889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.233.148:40727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:01:34.221825 28889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:34.224094 28889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:34.245075 28902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:34.245127 28900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:34.246682 28899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:34.251562 28889 server_base.cc:1034] running on GCE node
I20260924 16:01:34.252291 28889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:34.253413 28889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:34.256795 28889 hybrid_clock.cc:648] HybridClock initialized: now 1790265694256685 us; error 96 us; skew 500 ppm
I20260924 16:01:34.257190 28889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:01:34 dist-test-slave-xn4h krb5kdc[28323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265694, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:34.260972 28889 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:01:34.263392 28889 webserver.cc:460] Webserver started at http://127.16.233.129:37393/ using document root <none> and password file <none>
I20260924 16:01:34.264372 28889 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:34.264592 28889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:34.266912 28889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:01:34 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265694, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:01:34.306600 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "c5d8bc38b47446c483c286ef7c3d629b"
format_stamp: "Formatted at 2026-09-24 16:01:34 on dist-test-slave-xn4h"
server_key: "11551a2c4c25afe2695dc5a4af536f28"
server_key_iv: "cc831e0c3bd9749d90460a0d56a23d63"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:34.307936 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "c5d8bc38b47446c483c286ef7c3d629b"
format_stamp: "Formatted at 2026-09-24 16:01:34 on dist-test-slave-xn4h"
server_key: "11551a2c4c25afe2695dc5a4af536f28"
server_key_iv: "cc831e0c3bd9749d90460a0d56a23d63"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:34.320830 28889 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.000s
I20260924 16:01:34.365747 28909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:34.371523 28889 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.005s sys 0.012s
I20260924 16:01:34.371865 28889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "c5d8bc38b47446c483c286ef7c3d629b"
format_stamp: "Formatted at 2026-09-24 16:01:34 on dist-test-slave-xn4h"
server_key: "11551a2c4c25afe2695dc5a4af536f28"
server_key_iv: "cc831e0c3bd9749d90460a0d56a23d63"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:34.372267 28889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:34.417407 28889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:34.422372 28889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:34.422708 28889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:34.424227 28889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:01:34.427747 28889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:34.427861 28889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:34.428025 28889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:34.428109 28889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:01:34 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265693, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:01:34.554013 28889 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:40901
I20260924 16:01:34.554035 29023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:40901 every 8 connection(s)
I20260924 16:01:34.556058 28889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:01:34.564695 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 28889
I20260924 16:01:34.564958 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:01:34.565934 17318 external_mini_cluster.cc:1428] Setting key 3b7f3006660f85c84377ef8e85794502
Sep 24 16:01:34 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265694, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 16:01:34.601627 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:34.559071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:55033 (local address 127.16.233.190:35743)
0924 16:01:34.574587 (+ 15516us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:34.574594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:34.574617 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:01:34.574684 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:34.574689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:34.574772 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:34.574882 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:34.580084 (+ 5202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:34.581505 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:34.585450 (+ 3945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:34.586201 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:34.586391 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:34.591585 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:34.591592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:34.591597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:34.591657 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:34.595497 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:34.597003 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:34.597010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:34.597014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:34.597355 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:34.598340 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:34.598348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:34.598352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:34.598984 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:34.599122 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:34.600721 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0924 16:01:34.601028 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15339,"thread_start_us":15037,"threads_started":1}
I20260924 16:01:34.603694 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:34.559679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:35743 (local address 127.16.233.129:55033)
0924 16:01:34.569243 (+ 9564us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:34.569313 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:34.572210 (+ 2897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:34.576550 (+ 4340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:34.576569 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:34.577759 (+ 1190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:34.579732 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:34.579766 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:34.581827 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:34.581833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:34.584667 (+ 2834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:34.584711 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:34.586911 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:34.586921 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:34.587333 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:34.588330 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:34.588367 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:34.591273 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:34.595755 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:34.595767 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:34.595789 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:34.596836 (+ 1047us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:34.597543 (+ 707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:34.597548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:34.597551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:34.598137 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:34.599454 (+ 1317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:34.599468 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:34.600067 (+ 599us) client_negotiation.cc:769] Sending connection context
0924 16:01:34.601452 (+ 1385us) client_negotiation.cc:240] Negotiation successful
0924 16:01:34.602186 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9182,"thread_start_us":12891,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
I20260924 16:01:34.606293 29024 heartbeater.cc:347] Connected to a master server at 127.16.233.190:35743
I20260924 16:01:34.609829 29024 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:34.611461 29024 heartbeater.cc:511] Master 127.16.233.190:35743 requested a full tablet report, sending...
I20260924 16:01:34.614702 28807 ts_manager.cc:194] Registered new tserver with Master: c5d8bc38b47446c483c286ef7c3d629b (127.16.233.129:40901)
I20260924 16:01:34.616955 28807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:55033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 16:01:34.646086 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:40727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35743
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:01:34 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265693, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
W20260924 16:01:35.233943 29030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:35.234270 29030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:35.234393 29030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:35.241971 29030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:35.242092 29030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:35.242206 29030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:01:35.253827 29030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.233.148:40727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:01:35.255512 29030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:35.257009 29030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:35.273068 29044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:35.273209 29045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:35.273866 29047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:35.274696 29030 server_base.cc:1034] running on GCE node
I20260924 16:01:35.275468 29030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:35.276682 29030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:35.277987 29030 hybrid_clock.cc:648] HybridClock initialized: now 1790265695277811 us; error 90 us; skew 500 ppm
I20260924 16:01:35.278651 29030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:01:35 dist-test-slave-xn4h krb5kdc[28323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265695, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:35.282759 29030 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:01:35.284885 29030 webserver.cc:460] Webserver started at http://127.16.233.130:35603/ using document root <none> and password file <none>
I20260924 16:01:35.285856 29030 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:35.286010 29030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:35.286510 29030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:01:35 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265695, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:01:35.304397 29030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "83590092353544f7af0c3715c93717b4"
format_stamp: "Formatted at 2026-09-24 16:01:35 on dist-test-slave-xn4h"
server_key: "31722547d666c94c3bd2cc5cdbfe5bd0"
server_key_iv: "0d65436928f412cdc47195b9831e52b7"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:35.305650 29030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "83590092353544f7af0c3715c93717b4"
format_stamp: "Formatted at 2026-09-24 16:01:35 on dist-test-slave-xn4h"
server_key: "31722547d666c94c3bd2cc5cdbfe5bd0"
server_key_iv: "0d65436928f412cdc47195b9831e52b7"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:35.314011 29030 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260924 16:01:35.331410 29054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:35.333014 29030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:01:35.333223 29030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "83590092353544f7af0c3715c93717b4"
format_stamp: "Formatted at 2026-09-24 16:01:35 on dist-test-slave-xn4h"
server_key: "31722547d666c94c3bd2cc5cdbfe5bd0"
server_key_iv: "0d65436928f412cdc47195b9831e52b7"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:35.333473 29030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:35.354087 29030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:35.359853 29030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:35.360245 29030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:35.361373 29030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:01:35.363287 29030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:35.363399 29030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:35.363692 29030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:35.363873 29030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:35.425495 29030 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:37369
I20260924 16:01:35.425531 29168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:37369 every 8 connection(s)
I20260924 16:01:35.426956 29030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:01:35.432614 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 29030
I20260924 16:01:35.432806 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:01:35.433370 17318 external_mini_cluster.cc:1428] Setting key 1b580f6dfc4ce36611f8e676f1d471fa
Sep 24 16:01:35 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265695, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:01:35.449617 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:35.429601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:52185 (local address 127.16.233.190:35743)
0924 16:01:35.430051 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:35.430056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:35.431138 (+ 1082us) server_negotiation.cc:407] Connection header received
0924 16:01:35.432999 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:35.433005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:35.433104 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:35.433220 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:35.434783 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:35.435568 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:35.437158 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:35.437691 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:35.437828 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:35.441608 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:35.441614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:35.441617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:35.441651 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:35.445721 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:35.446626 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:35.446631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:35.446635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:35.446850 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:35.447339 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:35.447345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:35.447349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:35.447735 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:35.447949 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:35.448766 (+ 817us) server_negotiation.cc:299] Negotiation successful
0924 16:01:35.449053 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260924 16:01:35.450703 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:35.430200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:35743 (local address 127.16.233.130:52185)
0924 16:01:35.430968 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:35.431011 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:35.432617 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:35.433467 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:35.433480 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:35.433956 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:35.434597 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:35.434617 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:35.435749 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:35.435754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:35.436997 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:35.437010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:35.437931 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:35.437941 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:35.438210 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:35.439026 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:35.439051 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:35.441357 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:35.445892 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:35.445903 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:35.445922 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:35.446488 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:35.446977 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:35.446983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:35.446986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:35.447214 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:35.447877 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:35.447900 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:35.448295 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 16:01:35.448572 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 16:01:35.449248 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":288,"threads_started":1}
I20260924 16:01:35.452673 29169 heartbeater.cc:347] Connected to a master server at 127.16.233.190:35743
I20260924 16:01:35.453120 29169 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:35.454397 29169 heartbeater.cc:511] Master 127.16.233.190:35743 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 16:01:35.456107 28807 ts_manager.cc:194] Registered new tserver with Master: 83590092353544f7af0c3715c93717b4 (127.16.233.130:37369)
I20260924 16:01:35.457027 28807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:52185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 16:01:35.486111 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:40727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35743
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:01:35.629510 29024 heartbeater.cc:503] Master 127.16.233.190:35743 was elected leader, sending a full tablet report...
W20260924 16:01:35.918977 29175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:35.919291 29175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:35.919391 29175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:35.927500 29175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:35.927657 29175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:35.927907 29175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:01:35.938280 29175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.233.148:40727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:01:35.939888 29175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:35.941193 29175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:35.962340 29180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:35.962767 29183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:35.962879 29181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:35.964677 29175 server_base.cc:1034] running on GCE node
I20260924 16:01:35.965413 29175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:35.966707 29175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:35.968043 29175 hybrid_clock.cc:648] HybridClock initialized: now 1790265695967915 us; error 63 us; skew 500 ppm
I20260924 16:01:35.968503 29175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:01:35 dist-test-slave-xn4h krb5kdc[28323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265695, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:35.971908 29175 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:01:35.973661 29175 webserver.cc:460] Webserver started at http://127.16.233.131:37249/ using document root <none> and password file <none>
I20260924 16:01:35.974525 29175 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:35.974682 29175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:35.975026 29175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:01:35 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265695, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:01:35.993341 29175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "c3e150eb24704d56801138eb57053c14"
format_stamp: "Formatted at 2026-09-24 16:01:35 on dist-test-slave-xn4h"
server_key: "b51d630d5d528fd87ee7cfbd3cc4526c"
server_key_iv: "1e80e91f2ca38466f02ce4ed030a6bcd"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:35.994982 29175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "c3e150eb24704d56801138eb57053c14"
format_stamp: "Formatted at 2026-09-24 16:01:35 on dist-test-slave-xn4h"
server_key: "b51d630d5d528fd87ee7cfbd3cc4526c"
server_key_iv: "1e80e91f2ca38466f02ce4ed030a6bcd"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:36.001786 29175 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 16:01:36.020247 29190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:36.021968 29175 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 16:01:36.022172 29175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "c3e150eb24704d56801138eb57053c14"
format_stamp: "Formatted at 2026-09-24 16:01:35 on dist-test-slave-xn4h"
server_key: "b51d630d5d528fd87ee7cfbd3cc4526c"
server_key_iv: "1e80e91f2ca38466f02ce4ed030a6bcd"
server_key_version: "kuduclusterkey@0"
I20260924 16:01:36.022462 29175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:36.049827 29175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:36.055970 29175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:36.056520 29175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:36.058357 29175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:01:36.060410 29175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:36.060490 29175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:36.060645 29175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:36.060702 29175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:36.113478 29175 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:40301
I20260924 16:01:36.113617 29304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:40301 every 8 connection(s)
I20260924 16:01:36.115140 29175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:01:36.123068 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 29175
I20260924 16:01:36.123266 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:01:36.123747 17318 external_mini_cluster.cc:1428] Setting key 9f3749277778a5f254cde59716ee7846
Sep 24 16:01:36 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265695, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:01:36.145064 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.118597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:50319 (local address 127.16.233.190:35743)
0924 16:01:36.119806 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:36.119812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:36.120804 (+ 992us) server_negotiation.cc:407] Connection header received
0924 16:01:36.126524 (+ 5720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:36.126530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:36.126663 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:36.126836 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:36.129238 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.129988 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.133290 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.134041 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.134266 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.138943 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:36.138949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:36.138952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:36.138990 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:36.141636 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:36.142559 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:36.142565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:36.142570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:36.142710 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:36.143241 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:36.143249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:36.143256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:36.143538 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:36.143702 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:36.144455 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 16:01:36.144690 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":787,"threads_started":1}
I20260924 16:01:36.146098 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.119545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:35743 (local address 127.16.233.131:50319)
0924 16:01:36.120628 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:36.120679 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:36.124930 (+ 4251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:36.127227 (+ 2297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:36.127246 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:36.128046 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:36.129032 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.129060 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.130164 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.130178 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.133088 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.133126 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.134388 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.134397 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.134758 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.135650 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:36.135676 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:36.138653 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:36.141813 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:36.141823 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:36.141843 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:36.142421 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:36.142831 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:36.142837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:36.142840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:36.143074 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:36.143694 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:36.143703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:36.144064 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 16:01:36.144345 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 16:01:36.144915 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":420,"threads_started":1}
I20260924 16:01:36.147831 29305 heartbeater.cc:347] Connected to a master server at 127.16.233.190:35743
I20260924 16:01:36.148164 29305 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:36.148901 29305 heartbeater.cc:511] Master 127.16.233.190:35743 requested a full tablet report, sending...
I20260924 16:01:36.150463 28807 ts_manager.cc:194] Registered new tserver with Master: c3e150eb24704d56801138eb57053c14 (127.16.233.131:40301)
I20260924 16:01:36.151383 28807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:50319
I20260924 16:01:36.160950 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:01:36.184208 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.166745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.16.233.190:35743)
0924 16:01:36.167014 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:36.167021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:36.168087 (+ 1066us) server_negotiation.cc:407] Connection header received
0924 16:01:36.168264 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:36.168269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:36.168399 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:36.168493 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:36.170862 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.171846 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.173722 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.174544 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.174663 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.178058 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:36.178064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:36.178068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:36.178108 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:36.181345 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:36.182302 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:36.182306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:36.182309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:36.182456 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:36.182859 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:36.182862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:36.182864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:36.183157 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:36.183354 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:36.183736 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 16:01:36.183916 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:01:36.188668 28807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:01:36.198079 28807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:01:36.238636 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.214491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:40301 (local address 127.0.0.1:60056)
0924 16:01:36.230476 (+ 15985us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:36.230499 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:36.230659 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:36.231464 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:36.231469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:36.231496 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:36.231942 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.231954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.233992 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.234000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.236145 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.236158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.237600 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.237606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.237802 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.237845 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 16:01:36.237984 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:01:36.238133 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15836,"mutex_wait_us":70,"thread_start_us":12469,"threads_started":1}
I20260924 16:01:36.239090 29319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.214396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.16.233.131:40301)
0924 16:01:36.225525 (+ 11129us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:36.225538 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:36.230634 (+ 5096us) server_negotiation.cc:407] Connection header received
0924 16:01:36.230901 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:36.230914 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:36.231246 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:36.231484 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:36.232352 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.233860 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.236330 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.237375 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.237585 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.237800 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:36.238100 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 16:01:36.238282 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10900,"thread_start_us":10723,"threads_started":1}
I20260924 16:01:36.239225 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.214301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.16.233.129:40901)
0924 16:01:36.228294 (+ 13993us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:36.228306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:36.228346 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:01:36.228483 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:36.228496 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:36.228895 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:36.229178 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:36.230593 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.232403 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.235379 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.237008 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.238420 (+ 1412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.238615 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:36.238707 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 16:01:36.238873 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13718,"thread_start_us":13545,"threads_started":1}
I20260924 16:01:36.239715 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.214067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.0.0.1:35074)
0924 16:01:36.226360 (+ 12293us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:36.226421 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:36.226686 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:36.229311 (+ 2625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:36.229324 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:36.229387 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:36.230108 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.230131 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.232562 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.232567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.235176 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.235192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.237393 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.237400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.237655 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.237722 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 16:01:36.237838 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:01:36.238063 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11902,"mutex_wait_us":3,"thread_start_us":16845,"threads_started":1}
I20260924 16:01:36.240644 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.217906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.16.233.130:37369)
0924 16:01:36.229902 (+ 11996us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:36.229915 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:36.231376 (+ 1461us) server_negotiation.cc:407] Connection header received
0924 16:01:36.232359 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:36.232367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:36.232665 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:36.232888 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:36.233592 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.235150 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.237660 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.239169 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.239315 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.239488 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:36.240189 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 16:01:36.240360 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11821,"thread_start_us":11709,"threads_started":1}
I20260924 16:01:36.242468 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.217349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:37369 (local address 127.0.0.1:59604)
0924 16:01:36.231247 (+ 13898us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:36.231266 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:36.232165 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:36.232899 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:36.232904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:36.232929 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:36.233374 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.233387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.235329 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.235335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.237456 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.237469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.239803 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.239813 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.239988 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.240041 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 16:01:36.240144 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 16:01:36.240289 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13757,"mutex_wait_us":1,"thread_start_us":12778,"threads_started":1}
I20260924 16:01:36.243827 29238 tablet_service.cc:1467] Processing CreateTablet for tablet e7a6a60dbf574e80a13ae624e3c446f7 (DEFAULT_TABLE table=test-table [id=70e280ec2a1544bf915f34d1bb9eeb77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:36.244467 28957 tablet_service.cc:1467] Processing CreateTablet for tablet e7a6a60dbf574e80a13ae624e3c446f7 (DEFAULT_TABLE table=test-table [id=70e280ec2a1544bf915f34d1bb9eeb77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:36.245922 29238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a6a60dbf574e80a13ae624e3c446f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:36.246081 28957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a6a60dbf574e80a13ae624e3c446f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:36.245776 29102 tablet_service.cc:1467] Processing CreateTablet for tablet e7a6a60dbf574e80a13ae624e3c446f7 (DEFAULT_TABLE table=test-table [id=70e280ec2a1544bf915f34d1bb9eeb77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:36.247100 29102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a6a60dbf574e80a13ae624e3c446f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:36.259483 29324 tablet_bootstrap.cc:492] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: Bootstrap starting.
I20260924 16:01:36.264478 29324 tablet_bootstrap.cc:654] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:36.265861 29324 log.cc:824] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:36.269328 29324 tablet_bootstrap.cc:492] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: No bootstrap required, opened a new log
I20260924 16:01:36.269654 29324 ts_tablet_manager.cc:1397] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260924 16:01:36.271194 29326 tablet_bootstrap.cc:492] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: Bootstrap starting.
I20260924 16:01:36.274492 29325 tablet_bootstrap.cc:492] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: Bootstrap starting.
I20260924 16:01:36.276780 29326 tablet_bootstrap.cc:654] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:36.279206 29325 tablet_bootstrap.cc:654] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:36.279412 29326 log.cc:824] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:36.280752 29324 raft_consensus.cc:348] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } }
I20260924 16:01:36.281172 29325 log.cc:824] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:36.281281 29324 raft_consensus.cc:374] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:36.281451 29324 raft_consensus.cc:729] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83590092353544f7af0c3715c93717b4, State: Initialized, Role: FOLLOWER
I20260924 16:01:36.282166 29324 consensus_queue.cc:260] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } }
I20260924 16:01:36.284096 29326 tablet_bootstrap.cc:492] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: No bootstrap required, opened a new log
I20260924 16:01:36.284293 29324 ts_tablet_manager.cc:1428] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260924 16:01:36.284607 29326 ts_tablet_manager.cc:1397] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260924 16:01:36.285575 29169 heartbeater.cc:503] Master 127.16.233.190:35743 was elected leader, sending a full tablet report...
I20260924 16:01:36.285764 29325 tablet_bootstrap.cc:492] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: No bootstrap required, opened a new log
I20260924 16:01:36.286128 29325 ts_tablet_manager.cc:1397] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20260924 16:01:36.298530 29326 raft_consensus.cc:348] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } }
I20260924 16:01:36.299124 29326 raft_consensus.cc:374] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:36.299252 29326 raft_consensus.cc:729] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e150eb24704d56801138eb57053c14, State: Initialized, Role: FOLLOWER
I20260924 16:01:36.299319 29325 raft_consensus.cc:348] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } }
I20260924 16:01:36.299858 29325 raft_consensus.cc:374] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:36.299994 29325 raft_consensus.cc:729] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d8bc38b47446c483c286ef7c3d629b, State: Initialized, Role: FOLLOWER
I20260924 16:01:36.300040 29326 consensus_queue.cc:260] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } }
I20260924 16:01:36.300674 29325 consensus_queue.cc:260] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } }
I20260924 16:01:36.301534 29305 heartbeater.cc:503] Master 127.16.233.190:35743 was elected leader, sending a full tablet report...
I20260924 16:01:36.302367 29326 ts_tablet_manager.cc:1428] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260924 16:01:36.303915 29325 ts_tablet_manager.cc:1428] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
W20260924 16:01:36.331118 29025 tablet.cc:2369] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:01:36.367666 29306 tablet.cc:2369] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:01:36.429284 29170 tablet.cc:2369] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:01:36.701071 29330 raft_consensus.cc:482] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:01:36.701385 29330 raft_consensus.cc:504] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } }
I20260924 16:01:36.703238 29330 leader_election.cc:288] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5d8bc38b47446c483c286ef7c3d629b (127.16.233.129:40901), c3e150eb24704d56801138eb57053c14 (127.16.233.131:40301)
I20260924 16:01:36.710173 29319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.703803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:53535 (local address 127.16.233.131:40301)
0924 16:01:36.703985 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:36.703993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:36.704506 (+ 513us) server_negotiation.cc:407] Connection header received
0924 16:01:36.704737 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:36.704746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:36.704843 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:36.704944 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:36.705560 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.706399 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.708366 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.709180 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.709368 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.709454 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:36.709765 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 16:01:36.709900 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:01:36.710348 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.703597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:40301 (local address 127.16.233.130:53535)
0924 16:01:36.704384 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:36.704401 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:36.704583 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:36.705006 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:36.705013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:36.705058 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:36.705383 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.705393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.706541 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.706546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.708203 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.708216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.709405 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.709411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.709584 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.709616 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:01:36.709723 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:01:36.709868 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":24,"thread_start_us":454,"threads_started":1}
I20260924 16:01:36.710841 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.703628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.16.233.130:35625)
0924 16:01:36.705155 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:36.705177 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:36.705328 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:36.705677 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:36.705682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:36.705715 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:36.706075 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.706084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.707306 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.707313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.708767 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:36.708780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.709990 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.709996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:36.710155 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.710182 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:01:36.710296 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 16:01:36.710451 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1392,"mutex_wait_us":66,"thread_start_us":1138,"threads_started":1}
I20260924 16:01:36.711208 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.703597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:35625 (local address 127.16.233.129:40901)
0924 16:01:36.703847 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:36.703854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:36.705276 (+ 1422us) server_negotiation.cc:407] Connection header received
0924 16:01:36.705488 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:36.705492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:36.705565 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:36.705663 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:36.706240 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.707164 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.708925 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.709735 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.709916 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.710002 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:36.710762 (+ 760us) server_negotiation.cc:299] Negotiation successful
0924 16:01:36.710957 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:01:36.711433 29258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a6a60dbf574e80a13ae624e3c446f7" candidate_uuid: "83590092353544f7af0c3715c93717b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e150eb24704d56801138eb57053c14" is_pre_election: true
I20260924 16:01:36.711928 29258 raft_consensus.cc:2454] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83590092353544f7af0c3715c93717b4 in term 0.
I20260924 16:01:36.712170 28978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a6a60dbf574e80a13ae624e3c446f7" candidate_uuid: "83590092353544f7af0c3715c93717b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8bc38b47446c483c286ef7c3d629b" is_pre_election: true
I20260924 16:01:36.712590 29058 leader_election.cc:302] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83590092353544f7af0c3715c93717b4, c3e150eb24704d56801138eb57053c14; no voters:
I20260924 16:01:36.712733 28978 raft_consensus.cc:2454] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83590092353544f7af0c3715c93717b4 in term 0.
I20260924 16:01:36.713057 29330 raft_consensus.cc:2781] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:01:36.713263 29330 raft_consensus.cc:482] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:01:36.713389 29330 raft_consensus.cc:3037] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:36.715663 29330 raft_consensus.cc:504] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } }
I20260924 16:01:36.716285 29330 leader_election.cc:288] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [CANDIDATE]: Term 1 election: Requested vote from peers c5d8bc38b47446c483c286ef7c3d629b (127.16.233.129:40901), c3e150eb24704d56801138eb57053c14 (127.16.233.131:40301)
I20260924 16:01:36.716817 29258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a6a60dbf574e80a13ae624e3c446f7" candidate_uuid: "83590092353544f7af0c3715c93717b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e150eb24704d56801138eb57053c14"
I20260924 16:01:36.716845 28978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a6a60dbf574e80a13ae624e3c446f7" candidate_uuid: "83590092353544f7af0c3715c93717b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8bc38b47446c483c286ef7c3d629b"
I20260924 16:01:36.717042 29258 raft_consensus.cc:3037] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:36.717058 28978 raft_consensus.cc:3037] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:36.719522 29258 raft_consensus.cc:2454] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83590092353544f7af0c3715c93717b4 in term 1.
I20260924 16:01:36.719785 28978 raft_consensus.cc:2454] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83590092353544f7af0c3715c93717b4 in term 1.
I20260924 16:01:36.720073 29058 leader_election.cc:302] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83590092353544f7af0c3715c93717b4, c3e150eb24704d56801138eb57053c14; no voters:
I20260924 16:01:36.720444 29330 raft_consensus.cc:2781] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:36.720839 29330 raft_consensus.cc:686] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 1 LEADER]: Becoming Leader. State: Replica: 83590092353544f7af0c3715c93717b4, State: Running, Role: LEADER
I20260924 16:01:36.721340 29330 consensus_queue.cc:237] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } }
I20260924 16:01:36.726524 28807 catalog_manager.cc:5495] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83590092353544f7af0c3715c93717b4 (127.16.233.130). New cstate: current_term: 1 leader_uuid: "83590092353544f7af0c3715c93717b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83590092353544f7af0c3715c93717b4" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 37369 } health_report { overall_health: HEALTHY } } }
I20260924 16:01:36.816821 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.810625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.16.233.130:37369)
0924 16:01:36.811447 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:36.811452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:36.811469 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:01:36.811564 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:36.811568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:36.811657 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:36.811807 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:36.812273 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.812975 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.814655 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.815071 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.815177 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.815658 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:36.816197 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:36.816313 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:36.816394 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 16:01:36.816465 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":615,"threads_started":1}
I20260924 16:01:36.825714 29258 raft_consensus.cc:1261] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [term 1 FOLLOWER]: Refusing update from remote peer 83590092353544f7af0c3715c93717b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:36.825767 28978 raft_consensus.cc:1261] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [term 1 FOLLOWER]: Refusing update from remote peer 83590092353544f7af0c3715c93717b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:36.826930 29335 consensus_queue.cc:1035] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e150eb24704d56801138eb57053c14" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:36.827344 29330 consensus_queue.cc:1035] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d8bc38b47446c483c286ef7c3d629b" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 40901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:36.839363 29343 mvcc.cc:205] Tried to move back new op lower bound from 7332928294187360256 to 7332928293774241792. Current Snapshot: MvccSnapshot[applied={T|T < 7332928294187360256}]
I20260924 16:01:36.842831 29347 mvcc.cc:205] Tried to move back new op lower bound from 7332928294187360256 to 7332928293774241792. Current Snapshot: MvccSnapshot[applied={T|T < 7332928294187360256}]
I20260924 16:01:36.855041 28807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:01:36.855330 28807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:01:36.859355 28807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b2ece20d6084911a2f1aa041fbc1cf5: Sending DeleteTablet for 3 replicas of tablet e7a6a60dbf574e80a13ae624e3c446f7
I20260924 16:01:36.860881 29102 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a6a60dbf574e80a13ae624e3c446f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:36 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:59604
I20260924 16:01:36.860817 28957 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a6a60dbf574e80a13ae624e3c446f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:36 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:35074
I20260924 16:01:36.862116 29238 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a6a60dbf574e80a13ae624e3c446f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:36 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:60056
I20260924 16:01:36.864207 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 28889
I20260924 16:01:36.871796 29356 tablet_replica.cc:330] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: stopping tablet replica
I20260924 16:01:36.873315 29356 raft_consensus.cc:2229] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:01:36.874962 29356 raft_consensus.cc:2258] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:36.875818 29355 tablet_replica.cc:330] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: stopping tablet replica
I20260924 16:01:36.876935 29355 raft_consensus.cc:2229] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:01:36.878978 29355 raft_consensus.cc:2258] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:36.880200 29356 ts_tablet_manager.cc:1905] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:36.888031 29354 tablet_replica.cc:330] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: stopping tablet replica
I20260924 16:01:36.888203 29355 ts_tablet_manager.cc:1905] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:36.888914 29354 raft_consensus.cc:2229] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:01:36.889853 29354 raft_consensus.cc:2258] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 16:01:36 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 16:01:36.896926 29354 ts_tablet_manager.cc:1905] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:36.900524 29355 ts_tablet_manager.cc:1918] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:01:36.900826 29355 log.cc:1196] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/e7a6a60dbf574e80a13ae624e3c446f7
I20260924 16:01:36.901443 29355 ts_tablet_manager.cc:1939] T e7a6a60dbf574e80a13ae624e3c446f7 P 83590092353544f7af0c3715c93717b4: Deleting consensus metadata
I20260924 16:01:36.903692 28792 catalog_manager.cc:4841] TS 83590092353544f7af0c3715c93717b4 (127.16.233.130:37369): tablet e7a6a60dbf574e80a13ae624e3c446f7 (table test-table [id=70e280ec2a1544bf915f34d1bb9eeb77]) successfully deleted
I20260924 16:01:36.905074 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:36.865213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.16.233.129:40901)
0924 16:01:36.865411 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:36.865418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:36.878358 (+ 12940us) server_negotiation.cc:407] Connection header received
0924 16:01:36.879009 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:36.879015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:36.879118 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:36.879206 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:36.881322 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.882563 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.884731 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:36.885947 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:36.886068 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:36.893361 (+ 7293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:36.893383 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:36.893406 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:36.893463 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:36.900816 (+ 7353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:36.902070 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:36.902079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:36.902083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:36.902274 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:36.902833 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:36.902838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:36.902842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:36.903369 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:36.903729 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:36.904353 (+ 624us) server_negotiation.cc:299] Negotiation successful
0924 16:01:36.904654 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:01:36.906006 29019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:01:36.906464 29356 ts_tablet_manager.cc:1918] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:01:36.906726 29356 log.cc:1196] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/e7a6a60dbf574e80a13ae624e3c446f7
I20260924 16:01:36.907289 29356 ts_tablet_manager.cc:1939] T e7a6a60dbf574e80a13ae624e3c446f7 P c3e150eb24704d56801138eb57053c14: Deleting consensus metadata
I20260924 16:01:36.909389 28794 catalog_manager.cc:4841] TS c3e150eb24704d56801138eb57053c14 (127.16.233.131:40301): tablet e7a6a60dbf574e80a13ae624e3c446f7 (table test-table [id=70e280ec2a1544bf915f34d1bb9eeb77]) successfully deleted
I20260924 16:01:37.016355 29354 ts_tablet_manager.cc:1918] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:01:37.016712 29354 log.cc:1196] T e7a6a60dbf574e80a13ae624e3c446f7 P c5d8bc38b47446c483c286ef7c3d629b: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/e7a6a60dbf574e80a13ae624e3c446f7
I20260924 16:01:37.016954 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 28889
I20260924 16:01:37.035890 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 29030
W20260924 16:01:37.035960 28793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.129:40901 (error 108)
I20260924 16:01:37.040169 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.037420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.0.0.1:35092)
0924 16:01:37.039737 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:37.039919 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2181,"thread_start_us":2028,"threads_started":1}
W20260924 16:01:37.040470 28793 catalog_manager.cc:4584] TS c5d8bc38b47446c483c286ef7c3d629b (127.16.233.129:40901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7a6a60dbf574e80a13ae624e3c446f7: Network error: Client connection negotiation failed: client connection to 127.16.233.129:40901: connect: Connection refused (error 111)
Sep 24 16:01:37 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:01:37.055214 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.037135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.16.233.130:37369)
0924 16:01:37.037307 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:37.037313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:37.037334 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:01:37.037450 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:37.037456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:37.037532 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:37.037647 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:37.039319 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:37.040523 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:37.042547 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:37.042987 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:37.043100 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:37.048408 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:37.048425 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:37.048441 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:37.048492 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:37.052002 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:37.052840 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:37.052848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:37.052850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:37.052980 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:37.053398 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:37.053402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:37.053404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:37.053796 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:37.054155 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:37.054558 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 16:01:37.054833 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:01:37.055888 29164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:01:37.094224 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.093621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.0.0.1:35104)
0924 16:01:37.093888 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:37.094028 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 16:01:37.095553 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.095059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.0.0.1:35110)
0924 16:01:37.095247 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:37.095345 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 16:01:37.156250 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.155670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.0.0.1:35126)
0924 16:01:37.155930 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:37.156047 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 16:01:37.157437 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.157010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.0.0.1:35140)
0924 16:01:37.157182 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:37.157268 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 16:01:37.174201 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 29030
I20260924 16:01:37.193156 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 29175
Sep 24 16:01:37 dist-test-slave-xn4h krb5kdc[28323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 16:01:37.209407 29319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.193993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.16.233.131:40301)
0924 16:01:37.194189 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:37.194194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:37.194232 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:01:37.194292 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:37.194296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:37.194426 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:37.194520 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:37.195821 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:37.196588 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:37.197887 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:37.198494 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:37.198589 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:37.203473 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:37.203492 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:37.203509 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:37.203550 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:37.206633 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:37.207482 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:37.207489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:37.207492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:37.207609 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:37.207945 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:37.207947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:37.207949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:37.208307 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:37.208513 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:37.208869 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 16:01:37.209051 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:01:37.210217 29300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:01:37.238349 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.237767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.0.0.1:35150)
0924 16:01:37.238018 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:37.238136 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 16:01:37.239495 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.239091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.0.0.1:35154)
0924 16:01:37.239291 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:37.239364 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 16:01:37.302368 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 29175
I20260924 16:01:37.319448 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 28775
I20260924 16:01:37.320641 28838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:01:37.434088 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:37.433190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:40901 (local address 127.0.0.1:35162)
0924 16:01:37.433690 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:37.433837 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 16:01:37.434424 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 28775
I20260924 16:01:37.458213 17318 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:01:37.794416 17318 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:01:37.794682 17318 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:01:38.140540 17318 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:01:38.140 UTC [28354] LOG: received smart shutdown request
2026-09-24 16:01:38.144 UTC [28354] LOG: background worker "logical replication launcher" (PID 28363) exited with exit code 1
2026-09-24 16:01:38.145 UTC [28358] LOG: shutting down
2026-09-24 16:01:38.324 UTC [28354] LOG: database system is shut down
2026-09-24T16:01:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53620 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:01:38 dist-test-slave-xn4h krb5kdc[29372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:01:38 dist-test-slave-xn4h krb5kdc[29372](info): set up 2 sockets
Sep 24 16:01:38 dist-test-slave-xn4h krb5kdc[29372](info): commencing operation
krb5kdc: starting...
W20260924 16:01:39.576140 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:01:39 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:01:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:01:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/rangeradmin_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/rangeradmin_127.16.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/rangerlookup_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/rangerlookup_127.16.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/HTTP_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/HTTP_127.16.233.148@KRBTEST.COM.keytab.
I20260924 16:01:39.917452 17318 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRO1Jyr/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/postgres -l logfile start
2026-09-24 16:01:42.433 UTC [29403] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:01:42.433 UTC [29403] LOG: listening on IPv4 address "127.16.233.148", port 52733
2026-09-24 16:01:42.437 UTC [29403] LOG: listening on Unix socket "/tmp/.s.PGSQL.52733"
2026-09-24 16:01:42.460 UTC [29406] LOG: database system was shut down at 2026-09-24 16:01:41 UTC
2026-09-24 16:01:42.465 UTC [29403] LOG: database system is ready to accept connections
I20260924 16:01:43.447845 17318 mini_postgres.cc:96] Postgres bound to 52733
2026-09-24 16:01:43.465 UTC [29414] FATAL: database "slave" does not exist
127.16.233.148:52733 - accepting connections
I20260924 16:01:43.466298 17318 mini_ranger.cc:161] Starting Ranger...
I20260924 16:01:43.494280 17318 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:01:43.976287 17318 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:01:43.976389 17318 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-admin
2026-09-24 16:01:44,270 [I] DB FLAVOR :POSTGRES
2026-09-24 16:01:44,270 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:01:44,271 [I] Checking connection..
2026-09-24 16:01:44,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:01:44,527 [I] Checking connection passed.
2026-09-24 16:01:44,527 [I] --------- Verifying version history table ---------
2026-09-24 16:01:44,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:01:44,803 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:01:44,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:01:45,049 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:01:45,049 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:01:45,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:01:45.288 UTC [29478] WARNING: there is no transaction in progress
2026-09-24 16:01:45,295 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:01:45,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:01:45,561 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:01:45,561 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:01:45,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:01:45,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:01:46,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xn4h.c.gcp-upstream.internal','N') ;"
2026-09-24 16:01:46,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:01:46,613 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:01:46,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:01:46,882 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:01:46,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:01:47,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:01:47,378 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:01:47,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:01:48.392 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.409 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.427 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.442 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.458 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.476 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.531 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.539 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.553 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.564 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.575 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.590 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.600 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.606 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.613 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:48.620 UTC [29589] WARNING: there is no transaction in progress
2026-09-24 16:01:49,287 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:01:49,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:01:49,535 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:01:49,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:01:49,786 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:01:49,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:01:50,069 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:01:50,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:01:50,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:01:50,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xn4h.c.gcp-upstream.internal';"
2026-09-24 16:01:50,829 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:01:50,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:01:51,102 [I] DB_PATCHES have already been applied
I20260924 16:01:51.108578 17318 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRO1Jyr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:01:51.108681 17318 mini_ranger.cc:193] Using host: 127.16.233.148
I20260924 16:01:51.123888 17318 mini_ranger.cc:227] Ranger admin URL: http://127.16.233.148:43473
Sep 24, 2026 4:01:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:01:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:01:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:01:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:01:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:01:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:01:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:01:52.265444 17318 mini_ranger.cc:160] Time spent starting Ranger: real 8.799s user 0.000s sys 0.042s
Sep 24, 2026 4:02:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:02:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 4:02:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:02:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:02:13 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:02:13.810681 17318 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/rangerkms_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/rangerkms_127.16.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.233.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.233.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.233.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/HTTP_127.16.233.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.233.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/HTTP_127.16.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:02:13 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265733, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:02:13.987864 17318 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:02:14.009958 17318 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:02:14.922724 17318 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:02:14.922844 17318 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-kms
I20260924 16:02:14.922899 17318 mini_ranger_kms.cc:227] Using postgres at 127.16.233.148:52733
2026-09-24 16:02:15,292 [I] DB FLAVOR :POSTGRES
2026-09-24 16:02:15,292 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:02:15,292 [I] Checking connection
2026-09-24 16:02:15,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:02:15,565 [I] connection success
2026-09-24 16:02:15,565 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:02:15,565 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:02:15,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:02:15,823 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:02:15,824 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:02:15,824 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:02:15,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.148:52733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:02:16,093 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:02:16.608731 17318 mini_ranger_kms.cc:310] Created kms service
I20260924 16:02:16.811048 17318 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:02:16.861167 17318 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:02:16 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265733, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:17.271947 17318 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:02:17.272282 17318 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRO1Jyr/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRO1Jyr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:02:17.272349 17318 mini_ranger_kms.cc:240] Using host: 127.16.233.148
I20260924 16:02:17.287650 17318 mini_ranger_kms.cc:276] Ranger KMS PID: 29772
I20260924 16:02:17.287798 17318 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.233.148:58409
Sep 24, 2026 4:02:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:02:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:02:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:02:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:02:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:02:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 16:02:19.615177 17318 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.627s user 0.085s sys 0.302s
I20260924 16:02:19.615446 17318 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:02:19.615614 17318 mini_ranger_kms.cc:401] 127.16.233.148:58409/kms/v1/keys
Sep 24 16:02:19 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 16:02:20 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.148@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
Sep 24 16:02:20 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.148@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
Sep 24 16:02:22 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:02:23.054700 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45089
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:45089
--ranger_config_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:58409/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:23.399127 29823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:23.399479 29823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:23.399613 29823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:23.407099 29823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:23.407215 29823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:23.407260 29823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:23.407317 29823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:23.407366 29823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:23.417701 29823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.233.148:58409/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:45089
--ranger_config_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:23.419093 29823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:23.420423 29823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:23.433488 29829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:23.433522 29831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:23.433548 29828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:23.434022 29823 server_base.cc:1034] running on GCE node
I20260924 16:02:23.434978 29823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:23.436491 29823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:23.437697 29823 hybrid_clock.cc:648] HybridClock initialized: now 1790265743437641 us; error 50 us; skew 500 ppm
I20260924 16:02:23.438109 29823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:23 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265743, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:23.445042 29823 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:02:23.446880 29823 webserver.cc:460] Webserver started at http://127.16.233.190:41667/ using document root <none> and password file <none>
I20260924 16:02:23.447578 29823 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:23.447686 29823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:23.447965 29823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:02:23 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265743, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:23.500645 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "226905715b864d858142ac073514f6c7"
format_stamp: "Formatted at 2026-09-24 16:02:23 on dist-test-slave-xn4h"
server_key: "b2916330b9e3c562cc94d80f402f1d36"
server_key_iv: "fafc62e61d1c3566a8ac6182d20037ab"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:23.501972 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "226905715b864d858142ac073514f6c7"
format_stamp: "Formatted at 2026-09-24 16:02:23 on dist-test-slave-xn4h"
server_key: "b2916330b9e3c562cc94d80f402f1d36"
server_key_iv: "fafc62e61d1c3566a8ac6182d20037ab"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:23.509732 29823 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260924 16:02:23.531734 29838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:23.533469 29823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 16:02:23.533720 29823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "226905715b864d858142ac073514f6c7"
format_stamp: "Formatted at 2026-09-24 16:02:23 on dist-test-slave-xn4h"
server_key: "b2916330b9e3c562cc94d80f402f1d36"
server_key_iv: "fafc62e61d1c3566a8ac6182d20037ab"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:23.533969 29823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:23.547079 29823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:23.552309 29823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:23.552672 29823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:23.573730 29823 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:45089
I20260924 16:02:23.573731 29891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:45089 every 8 connection(s)
I20260924 16:02:23.575526 29823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:23.579875 29892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:23.579958 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 29823
I20260924 16:02:23.580176 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:02:23.581033 17318 external_mini_cluster.cc:1428] Setting key 98bb491a93c9ef48e6bef2256a05371c
I20260924 16:02:23.599654 29892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Bootstrap starting.
I20260924 16:02:23.603454 29892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:23.604838 29892 log.cc:824] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:23.608537 29892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: No bootstrap required, opened a new log
Sep 24 16:02:23 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:23.619407 29892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } }
I20260924 16:02:23.619915 29892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:23.620460 29892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226905715b864d858142ac073514f6c7, State: Initialized, Role: FOLLOWER
I20260924 16:02:23.621300 29892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } }
I20260924 16:02:23.621589 29892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:23.621743 29892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:23.621946 29892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:23.624989 29895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:23.583357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.16.233.190:45089)
0924 16:02:23.598585 (+ 15228us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:23.598604 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:23.598664 (+ 60us) server_negotiation.cc:407] Connection header received
0924 16:02:23.601276 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:23.601314 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:23.601896 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:23.602419 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:23.604419 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.606533 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.608509 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.609595 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.609800 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:23.615032 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:23.615044 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:23.615068 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:23.615151 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:23.619337 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.620346 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.620356 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.620367 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.620596 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.621083 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.621091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.621097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.621937 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:23.622286 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:23.622889 (+ 603us) server_negotiation.cc:299] Negotiation successful
0924 16:02:23.623362 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14900,"thread_start_us":14645,"threads_started":1}
I20260924 16:02:23.625452 29892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } }
I20260924 16:02:23.626129 29892 leader_election.cc:302] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 226905715b864d858142ac073514f6c7; no voters:
I20260924 16:02:23.627461 29892 leader_election.cc:288] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:23.627676 29897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:23.632474 29892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:23.634694 29897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 1 LEADER]: Becoming Leader. State: Replica: 226905715b864d858142ac073514f6c7, State: Running, Role: LEADER
I20260924 16:02:23.635358 29897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } }
I20260924 16:02:23.640007 29898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "226905715b864d858142ac073514f6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } } }
I20260924 16:02:23.640456 29898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:23.645704 29899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226905715b864d858142ac073514f6c7. Latest consensus state: current_term: 1 leader_uuid: "226905715b864d858142ac073514f6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } } }
I20260924 16:02:23.646049 29899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:23.646739 29908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:23.652693 29908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:23.653772 29892 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:02:23.680115 29908 catalog_manager.cc:1347] Generated new cluster ID: 46b3cf64e9a14805a6ca8340d8afa8c6
I20260924 16:02:23.680389 29908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:23.708201 29908 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:23.710343 29908 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:23.733881 29908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Generated new TSK 0
I20260924 16:02:23.735265 29908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 16:02:24.486696 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:58409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45089
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:02:24 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265744, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 16:02:24.948745 29938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:24.949254 29938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:24.949443 29938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:24.961580 29938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:24.961771 29938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:24.961997 29938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:02:24.975656 29938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.233.148:58409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:24.977607 29938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:24.979511 29938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:24.999609 29949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:25.003412 29948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:25.011121 29951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:25.011595 29938 server_base.cc:1034] running on GCE node
I20260924 16:02:25.012761 29938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:25.018680 29938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:25.020071 29938 hybrid_clock.cc:648] HybridClock initialized: now 1790265745019951 us; error 110 us; skew 500 ppm
I20260924 16:02:25.020504 29938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:25 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:25.025146 29938 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:02:25.029768 29938 webserver.cc:460] Webserver started at http://127.16.233.129:33645/ using document root <none> and password file <none>
I20260924 16:02:25.031028 29938 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:25.031209 29938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:25.031610 29938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:02:25 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:25.059741 29938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "2ee054632d45464fa24c5759f4e489ea"
format_stamp: "Formatted at 2026-09-24 16:02:25 on dist-test-slave-xn4h"
server_key: "d6505911a5e68693aa1a33b90c7993a3"
server_key_iv: "ab2b6de728225c8ce0c451f0c9aad993"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:25.061262 29938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "2ee054632d45464fa24c5759f4e489ea"
format_stamp: "Formatted at 2026-09-24 16:02:25 on dist-test-slave-xn4h"
server_key: "d6505911a5e68693aa1a33b90c7993a3"
server_key_iv: "ab2b6de728225c8ce0c451f0c9aad993"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:25.075582 29938 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.012s sys 0.000s
I20260924 16:02:25.109428 29959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:25.111646 29938 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.009s sys 0.002s
I20260924 16:02:25.111855 29938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "2ee054632d45464fa24c5759f4e489ea"
format_stamp: "Formatted at 2026-09-24 16:02:25 on dist-test-slave-xn4h"
server_key: "d6505911a5e68693aa1a33b90c7993a3"
server_key_iv: "ab2b6de728225c8ce0c451f0c9aad993"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:25.112159 29938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:25.137583 29938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:25.142352 29938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:25.142701 29938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:25.143837 29938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:25.145422 29938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:25.145503 29938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:25.145648 29938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:25.145704 29938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:25.262732 29938 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:46551
I20260924 16:02:25.262781 30075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:46551 every 8 connection(s)
I20260924 16:02:25.264576 29938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:25.275043 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 29938
I20260924 16:02:25.275311 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:25.276355 17318 external_mini_cluster.cc:1428] Setting key fc7a733b8fccacb9803019932653b989
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
Sep 24 16:02:25 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
I20260924 16:02:25.323705 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:25.267986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:38543 (local address 127.16.233.190:45089)
0924 16:02:25.296669 (+ 28683us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:25.296678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:25.299582 (+ 2904us) server_negotiation.cc:407] Connection header received
0924 16:02:25.301992 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:25.302002 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:25.302135 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:25.302239 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:25.304843 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:25.306197 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:25.309068 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:25.309899 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:25.310070 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:25.315111 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:25.315120 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:25.315126 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:25.315190 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:25.319016 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:25.320302 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:25.320311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:25.320317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:25.320477 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:25.321094 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:25.321100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:25.321103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:25.321525 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:25.321674 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:25.322893 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0924 16:02:25.323176 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28510,"thread_start_us":28376,"threads_started":1}
I20260924 16:02:25.326218 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:25.268305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45089 (local address 127.16.233.129:38543)
0924 16:02:25.297332 (+ 29027us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:25.297398 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:25.301582 (+ 4184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:25.302932 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:25.302948 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:25.303680 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:25.304604 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:25.304637 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:25.306472 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:25.306480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:25.308852 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:25.308871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:25.310493 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:25.310503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:25.310866 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:25.311736 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:25.311771 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:25.314762 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:25.319349 (+ 4587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:25.319361 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:25.319385 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:25.320132 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:25.320648 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:25.320655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:25.320662 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:25.320945 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:25.321793 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:25.321805 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:25.322383 (+ 578us) client_negotiation.cc:769] Sending connection context
0924 16:02:25.323992 (+ 1609us) client_negotiation.cc:240] Negotiation successful
0924 16:02:25.324655 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28639,"thread_start_us":30153,"threads_started":1}
I20260924 16:02:25.328325 30076 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45089
I20260924 16:02:25.328737 30076 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:25.329701 30076 heartbeater.cc:511] Master 127.16.233.190:45089 requested a full tablet report, sending...
I20260924 16:02:25.332633 29857 ts_manager.cc:194] Registered new tserver with Master: 2ee054632d45464fa24c5759f4e489ea (127.16.233.129:46551)
I20260924 16:02:25.334928 29857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:38543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 16:02:25.354607 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:58409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45089
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:02:25 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265744, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
Sep 24 16:02:25 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265744, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
W20260924 16:02:25.839581 30082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:25.840142 30082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:25.840368 30082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:25.852125 30082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:25.852387 30082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:25.852612 30082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:02:25.868450 30082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.233.148:58409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:25.870594 30082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:25.872467 30082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:25.906894 30089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:25.907419 30090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:25.925112 30082 server_base.cc:1034] running on GCE node
W20260924 16:02:25.924642 30092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:25.926397 30082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:25.933926 30082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:25.935317 30082 hybrid_clock.cc:648] HybridClock initialized: now 1790265745935209 us; error 79 us; skew 500 ppm
I20260924 16:02:25.935788 30082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:25 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:25.940336 30082 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:02:25.943203 30082 webserver.cc:460] Webserver started at http://127.16.233.130:40471/ using document root <none> and password file <none>
I20260924 16:02:25.944253 30082 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:25.944406 30082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:25.944823 30082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:02:25 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:25.964756 30082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "ab2fe64f33524f7e933b12621e372070"
format_stamp: "Formatted at 2026-09-24 16:02:25 on dist-test-slave-xn4h"
server_key: "f728506015b4d218e232da843cb51ed4"
server_key_iv: "f7d2ee45decee96b6b0518bc8792e69b"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:25.966635 30082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "ab2fe64f33524f7e933b12621e372070"
format_stamp: "Formatted at 2026-09-24 16:02:25 on dist-test-slave-xn4h"
server_key: "f728506015b4d218e232da843cb51ed4"
server_key_iv: "f7d2ee45decee96b6b0518bc8792e69b"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:25.991573 30082 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.007s sys 0.016s
I20260924 16:02:26.021370 30099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:26.023484 30082 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.005s
I20260924 16:02:26.023792 30082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "ab2fe64f33524f7e933b12621e372070"
format_stamp: "Formatted at 2026-09-24 16:02:25 on dist-test-slave-xn4h"
server_key: "f728506015b4d218e232da843cb51ed4"
server_key_iv: "f7d2ee45decee96b6b0518bc8792e69b"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:26.024135 30082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:26.079051 30082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:26.084748 30082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:26.085098 30082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:26.086388 30082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:26.088213 30082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:26.088382 30082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:02:26.088568 30082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:26.088677 30082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:26.154167 30082 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:41175
I20260924 16:02:26.154202 30220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:41175 every 8 connection(s)
I20260924 16:02:26.156054 30082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:26.164284 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 30082
I20260924 16:02:26.164464 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:26.164999 17318 external_mini_cluster.cc:1428] Setting key dd027a4a3f9ef832c818f0ae169f34fe
Sep 24 16:02:26 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:26.197958 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.161724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45089 (local address 127.16.233.130:60161)
0924 16:02:26.172938 (+ 11214us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.173005 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:26.175601 (+ 2596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:26.178137 (+ 2536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:26.178151 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:26.178851 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:26.179707 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.179736 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.181336 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.181341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.183478 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.183494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.184815 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.184824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.185170 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.186005 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:26.186036 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:26.188835 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:26.192857 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:26.192867 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:26.192886 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:26.193543 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:26.193981 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:26.193986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:26.193990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:26.194248 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:26.195034 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:26.195042 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:26.195517 (+ 475us) client_negotiation.cc:769] Sending connection context
0924 16:02:26.195879 (+ 362us) client_negotiation.cc:240] Negotiation successful
0924 16:02:26.196468 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10827,"thread_start_us":15117,"threads_started":1}
I20260924 16:02:26.199242 30223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.161235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:60161 (local address 127.16.233.190:45089)
0924 16:02:26.176512 (+ 15277us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.176519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.176541 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:02:26.176608 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.176613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.176700 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.176816 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:26.179949 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.181109 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.183686 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.184348 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.184474 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.189112 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:26.189119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:26.189136 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:26.189180 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:26.192673 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:26.193703 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:26.193709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:26.193712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:26.193853 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:26.194418 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:26.194423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:26.194426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:26.194814 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:26.194960 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.198632 (+ 3672us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.198885 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15120,"thread_start_us":15949,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 16:02:26.200748 30221 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45089
I20260924 16:02:26.201093 30221 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:26.201949 30221 heartbeater.cc:511] Master 127.16.233.190:45089 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
I20260924 16:02:26.203799 29857 ts_manager.cc:194] Registered new tserver with Master: ab2fe64f33524f7e933b12621e372070 (127.16.233.130:41175)
I20260924 16:02:26.204860 29857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:60161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 16:02:26.237914 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:58409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45089
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:26.339658 30076 heartbeater.cc:503] Master 127.16.233.190:45089 was elected leader, sending a full tablet report...
W20260924 16:02:26.640208 30227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:26.640540 30227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:26.640635 30227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:26.647927 30227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:26.648049 30227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:26.648165 30227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:02:26.658753 30227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.233.148:58409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:26.660104 30227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:26.661446 30227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:26.674796 30233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:26.674824 30232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:26.675256 30235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:26.675630 30227 server_base.cc:1034] running on GCE node
I20260924 16:02:26.676242 30227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:26.677105 30227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:26.678337 30227 hybrid_clock.cc:648] HybridClock initialized: now 1790265746678252 us; error 53 us; skew 500 ppm
I20260924 16:02:26.678663 30227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:26 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:26.681751 30227 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:02:26.683190 30227 webserver.cc:460] Webserver started at http://127.16.233.131:36049/ using document root <none> and password file <none>
I20260924 16:02:26.683840 30227 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:26.683939 30227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:26.684203 30227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:02:26 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:26.701588 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "d82a060cce054dc390c3755b49d9fe8e"
format_stamp: "Formatted at 2026-09-24 16:02:26 on dist-test-slave-xn4h"
server_key: "d85acdcd3396c0550320e5700f635b81"
server_key_iv: "3cef83873977dc743bb1b4faedb055e3"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:26.702539 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "d82a060cce054dc390c3755b49d9fe8e"
format_stamp: "Formatted at 2026-09-24 16:02:26 on dist-test-slave-xn4h"
server_key: "d85acdcd3396c0550320e5700f635b81"
server_key_iv: "3cef83873977dc743bb1b4faedb055e3"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:26.708189 30227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:02:26.725144 30242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:26.726684 30227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:26.726872 30227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "d82a060cce054dc390c3755b49d9fe8e"
format_stamp: "Formatted at 2026-09-24 16:02:26 on dist-test-slave-xn4h"
server_key: "d85acdcd3396c0550320e5700f635b81"
server_key_iv: "3cef83873977dc743bb1b4faedb055e3"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:26.727120 30227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:26.742944 30227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:26.747242 30227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:26.747555 30227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:26.748550 30227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:26.749922 30227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:26.749995 30227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:26.750172 30227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:26.750228 30227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:26.784108 30227 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:36241
I20260924 16:02:26.784143 30355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:36241 every 8 connection(s)
I20260924 16:02:26.785451 30227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:26.785902 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 30227
I20260924 16:02:26.786116 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:26.786782 17318 external_mini_cluster.cc:1428] Setting key f270e7e719bcea7f290acf5a254971ab
Sep 24 16:02:26 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:26.809360 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.788345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:50129 (local address 127.16.233.190:45089)
0924 16:02:26.789037 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.789041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.789934 (+ 893us) server_negotiation.cc:407] Connection header received
0924 16:02:26.791807 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.791810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.791886 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.791977 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:26.793306 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.794049 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.796893 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.797713 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.797862 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.802993 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:26.802999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:26.803002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:26.803036 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:26.806040 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:26.806915 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:26.806921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:26.806924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:26.807065 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:26.807537 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:26.807542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:26.807546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:26.807834 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:26.807996 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.808732 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.808969 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":275,"threads_started":1}
I20260924 16:02:26.810385 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.788908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45089 (local address 127.16.233.131:50129)
0924 16:02:26.789787 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.789830 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:26.791576 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:26.792187 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:26.792194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:26.792587 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:26.793158 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.793174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.794226 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.794231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.796682 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.796698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.798209 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.798219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.798592 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.799479 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:26.799517 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:26.802720 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:26.806207 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:26.806217 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:26.806235 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:26.806784 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:26.807193 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:26.807198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:26.807202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:26.807413 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:26.807975 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:26.807984 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:26.808346 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 16:02:26.808615 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 16:02:26.809158 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":376,"threads_started":1}
I20260924 16:02:26.812115 30356 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45089
I20260924 16:02:26.812434 30356 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:26.813146 30356 heartbeater.cc:511] Master 127.16.233.190:45089 requested a full tablet report, sending...
I20260924 16:02:26.814814 29857 ts_manager.cc:194] Registered new tserver with Master: d82a060cce054dc390c3755b49d9fe8e (127.16.233.131:36241)
I20260924 16:02:26.816212 29857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:50129
I20260924 16:02:26.816776 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:02:26.819186 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 29938
Sep 24 16:02:26 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 16:02:26.837419 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.820575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.16.233.129:46551)
0924 16:02:26.821193 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.821199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.821223 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:02:26.821280 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.821286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.821460 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.821621 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:26.823180 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.824186 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.825689 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.826240 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.826402 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.831038 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:26.831049 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:26.831064 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:26.831098 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:26.834519 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:26.835207 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:26.835216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:26.835224 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:26.835388 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:26.835796 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:26.835802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:26.835806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:26.836135 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:26.836472 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.836764 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.837053 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":369,"threads_started":1}
I20260924 16:02:26.839371 30069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:26.923250 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 29938
I20260924 16:02:26.940032 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 30082
Sep 24 16:02:26 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:02:26.957070 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.940945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.16.233.130:41175)
0924 16:02:26.941523 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.941537 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.941564 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:02:26.941625 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.941632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.941851 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.941984 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:26.943367 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.944394 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.945691 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.946282 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.946505 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.951237 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:26.951249 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:26.951264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:26.951299 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:26.954160 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:26.954898 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:26.954905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:26.954907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:26.955053 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:26.955401 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:26.955405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:26.955407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:26.955770 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:26.956011 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.956409 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.956636 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":310,"threads_started":1}
I20260924 16:02:26.958436 30216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:27.041283 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 30082
I20260924 16:02:27.057228 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 30227
Sep 24 16:02:27 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 16:02:27.074028 30365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:27.058246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.16.233.131:36241)
0924 16:02:27.058948 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:27.058955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:27.058984 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:02:27.059046 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:27.059053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:27.059289 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:27.059416 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:27.060805 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:27.061877 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:27.063407 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:27.063975 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:27.064097 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:27.068543 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:27.068556 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:27.068571 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:27.068615 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:27.071567 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:27.072250 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:27.072256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:27.072258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:27.072390 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:27.072703 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:27.072707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:27.072709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:27.073045 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:27.073298 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:27.073540 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 16:02:27.073739 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":411,"threads_started":1}
I20260924 16:02:27.075371 30351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:27.159296 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 30227
I20260924 16:02:27.176354 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 29823
I20260924 16:02:27.177699 29887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:27.285012 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 29823
I20260924 16:02:27.307691 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45089
--webserver_interface=127.16.233.190
--webserver_port=41667
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:45089
--ranger_config_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:58409/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.233.148:58409/kms with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:27.637225 30368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:27.637593 30368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:27.637754 30368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:27.645249 30368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:27.645429 30368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:27.645545 30368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:27.645646 30368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:27.645727 30368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:27.656149 30368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.233.148:58409/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:45089
--ranger_config_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=41667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:27.657490 30368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:27.658888 30368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:27.671566 30373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:27.671532 30376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:27.671497 30374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:27.672513 30368 server_base.cc:1034] running on GCE node
I20260924 16:02:27.673273 30368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:27.674775 30368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:27.675987 30368 hybrid_clock.cc:648] HybridClock initialized: now 1790265747675951 us; error 25 us; skew 500 ppm
I20260924 16:02:27.676318 30368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:27 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:27.679375 30368 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:02:27.680941 30368 webserver.cc:460] Webserver started at http://127.16.233.190:41667/ using document root <none> and password file <none>
I20260924 16:02:27.681607 30368 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:27.681757 30368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:02:27 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:27.745792 30368 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:02:27.749097 30384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:27.750633 30368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:27.750831 30368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "226905715b864d858142ac073514f6c7"
format_stamp: "Formatted at 2026-09-24 16:02:23 on dist-test-slave-xn4h"
server_key: "b2916330b9e3c562cc94d80f402f1d36"
server_key_iv: "fafc62e61d1c3566a8ac6182d20037ab"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:27.751825 30368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:27.772235 30368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:27.775959 30368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:27.776269 30368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:27.794872 30368 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:45089
I20260924 16:02:27.794891 30436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:45089 every 8 connection(s)
I20260924 16:02:27.796422 30368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:27.802815 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 30368
I20260924 16:02:27.803905 30437 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:02:27.803920 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:58409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:46551
--local_ip_for_outbound_sockets=127.16.233.129
--tserver_master_addrs=127.16.233.190:45089
--webserver_port=33645
--webserver_interface=127.16.233.129
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.233.148:58409/kms with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:27.809368 30437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Bootstrap starting.
I20260924 16:02:27.829237 30437 log.cc:824] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:27.839107 30437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:02:27.839691 30437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Bootstrap complete.
I20260924 16:02:27.848858 30437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } }
I20260924 16:02:27.849706 30437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226905715b864d858142ac073514f6c7, State: Initialized, Role: FOLLOWER
I20260924 16:02:27.850435 30437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } }
I20260924 16:02:27.850677 30437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:27.850816 30437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:27.851011 30437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:02:27.855063 30437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } }
I20260924 16:02:27.855646 30437 leader_election.cc:302] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 226905715b864d858142ac073514f6c7; no voters:
I20260924 16:02:27.856346 30437 leader_election.cc:288] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:02:27.856627 30441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:02:27.857756 30441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [term 2 LEADER]: Becoming Leader. State: Replica: 226905715b864d858142ac073514f6c7, State: Running, Role: LEADER
I20260924 16:02:27.858424 30441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } }
I20260924 16:02:27.858618 30437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:27.864122 30444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226905715b864d858142ac073514f6c7. Latest consensus state: current_term: 2 leader_uuid: "226905715b864d858142ac073514f6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } } }
I20260924 16:02:27.864495 30444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:27.865051 30442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "226905715b864d858142ac073514f6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226905715b864d858142ac073514f6c7" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45089 } } }
I20260924 16:02:27.865439 30442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:27.867659 30437 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:02:27.868418 30452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:27.891458 30452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:27.893308 30452 catalog_manager.cc:1259] Loaded cluster ID: 46b3cf64e9a14805a6ca8340d8afa8c6
I20260924 16:02:27.893541 30452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:27.898295 30452 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:27.900383 30452 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Loaded TSK: 0
I20260924 16:02:27.902040 30452 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:02:28.286947 30439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:28.287418 30439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:28.287603 30439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:28.298907 30439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:28.299077 30439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:28.299314 30439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:02:28.313252 30439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.233.148:58409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:46551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=33645
--tserver_master_addrs=127.16.233.190:45089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:28.315001 30439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:28.316339 30439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:28.357795 30478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:28.358338 30480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:28.358395 30476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:28.359887 30439 server_base.cc:1034] running on GCE node
I20260924 16:02:28.360718 30439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:28.364137 30439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:28.365430 30439 hybrid_clock.cc:648] HybridClock initialized: now 1790265748365363 us; error 64 us; skew 500 ppm
I20260924 16:02:28.365893 30439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:28 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:28.369391 30439 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:02:28.389240 30439 webserver.cc:460] Webserver started at http://127.16.233.129:33645/ using document root <none> and password file <none>
I20260924 16:02:28.390285 30439 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:28.390568 30439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:02:28 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:28.414934 30439 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 16:02:28.431067 30488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:28.434058 30439 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.010s sys 0.004s
I20260924 16:02:28.434381 30439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "2ee054632d45464fa24c5759f4e489ea"
format_stamp: "Formatted at 2026-09-24 16:02:25 on dist-test-slave-xn4h"
server_key: "d6505911a5e68693aa1a33b90c7993a3"
server_key_iv: "ab2b6de728225c8ce0c451f0c9aad993"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:28.435992 30439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:28.481390 30439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:28.487310 30439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:28.487699 30439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:28.489338 30439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:28.497478 30439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:28.497669 30439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:02:28.497830 30439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:28.497938 30439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:28.551342 30439 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:46551
I20260924 16:02:28.551369 30604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:46551 every 8 connection(s)
I20260924 16:02:28.553274 30439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:28.556026 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 30439
I20260924 16:02:28.556979 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:58409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:41175
--local_ip_for_outbound_sockets=127.16.233.130
--tserver_master_addrs=127.16.233.190:45089
--webserver_port=40471
--webserver_interface=127.16.233.130
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.233.148:58409/kms with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:02:28 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:28.600608 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.560320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45089 (local address 127.16.233.129:54827)
0924 16:02:28.567027 (+ 6707us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:28.567086 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:28.570404 (+ 3318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:28.572894 (+ 2490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:28.572908 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:28.573614 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:28.574496 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.574527 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.576984 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.576998 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.579309 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.579359 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.581414 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.581423 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.581753 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.583911 (+ 2158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:28.583951 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:28.589013 (+ 5062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:28.595146 (+ 6133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.595157 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.595177 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.595823 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.596410 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.596415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.596419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.596672 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.597817 (+ 1145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:28.597825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:28.598236 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 16:02:28.598568 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 16:02:28.599188 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6351,"thread_start_us":6096,"threads_started":1}
I20260924 16:02:28.600793 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.561884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:54827 (local address 127.16.233.190:45089)
0924 16:02:28.570893 (+ 9009us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:28.570914 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:28.570969 (+ 55us) server_negotiation.cc:407] Connection header received
0924 16:02:28.571483 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:28.571509 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:28.572084 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:28.572668 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:28.574763 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.576768 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.579557 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.580784 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.580992 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.590594 (+ 9602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:28.590608 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:28.590633 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:28.590694 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:28.594906 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.596043 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.596055 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.596063 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.596285 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.596829 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.596835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.596839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.597523 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:28.597733 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:28.598852 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0924 16:02:28.599360 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8713,"thread_start_us":8479,"threads_started":1}
I20260924 16:02:28.604984 30605 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45089
I20260924 16:02:28.605341 30605 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:28.606276 30605 heartbeater.cc:511] Master 127.16.233.190:45089 requested a full tablet report, sending...
W20260924 16:02:28.609594 30605 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:45089 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:02:28.609812 30605 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:28.610092 30605 heartbeater.cc:511] Master 127.16.233.190:45089 requested a full tablet report, sending...
I20260924 16:02:28.611534 30605 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:28.611838 30605 heartbeater.cc:511] Master 127.16.233.190:45089 requested a full tablet report, sending...
W20260924 16:02:28.613004 30605 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:02:28.953168 30609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:28.953500 30609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:28.953676 30609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:28.961577 30609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:28.961727 30609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:28.961899 30609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:02:28.973896 30609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.233.148:58409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:41175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=40471
--tserver_master_addrs=127.16.233.190:45089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:28.975890 30609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:28.977533 30609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:28.996666 30624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:28.998782 30623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:29.003170 30609 server_base.cc:1034] running on GCE node
W20260924 16:02:29.003486 30627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:29.004202 30609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:29.005381 30609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:29.006699 30609 hybrid_clock.cc:648] HybridClock initialized: now 1790265749006608 us; error 56 us; skew 500 ppm
I20260924 16:02:29.007048 30609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:29 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:29.010857 30609 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:02:29.013007 30609 webserver.cc:460] Webserver started at http://127.16.233.130:40471/ using document root <none> and password file <none>
I20260924 16:02:29.014027 30609 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:29.014196 30609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:02:29 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:29.038735 30609 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:02:29.044869 30634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:29.046599 30609 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s
I20260924 16:02:29.046893 30609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "ab2fe64f33524f7e933b12621e372070"
format_stamp: "Formatted at 2026-09-24 16:02:25 on dist-test-slave-xn4h"
server_key: "f728506015b4d218e232da843cb51ed4"
server_key_iv: "f7d2ee45decee96b6b0518bc8792e69b"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:29.048446 30609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:29.115593 30609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:29.121977 30609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:29.122411 30609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:29.129659 30609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:29.139804 30609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:29.140024 30609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:02:29.140223 30609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:29.140328 30609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:02:29 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:29.245428 30609 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:41175
I20260924 16:02:29.245477 30751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:41175 every 8 connection(s)
I20260924 16:02:29.247211 30609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:29.252054 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 30609
I20260924 16:02:29.252902 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.148:58409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:36241
--local_ip_for_outbound_sockets=127.16.233.131
--tserver_master_addrs=127.16.233.190:45089
--webserver_port=36049
--webserver_interface=127.16.233.131
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.233.148:58409/kms with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:02:29 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:29.282523 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.250702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:57285 (local address 127.16.233.190:45089)
0924 16:02:29.258732 (+ 8030us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.258739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.264325 (+ 5586us) server_negotiation.cc:407] Connection header received
0924 16:02:29.266557 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.266563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.266662 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.266783 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:29.268370 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.269284 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.271520 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.272186 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.272293 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.276423 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:29.276431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:29.276435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:29.276496 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:29.279283 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.280065 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.280071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.280079 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.280203 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.280672 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.280677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.280679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.280964 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:29.281122 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.281935 (+ 813us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.282149 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7856,"thread_start_us":7992,"threads_started":1}
I20260924 16:02:29.284593 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.251196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45089 (local address 127.16.233.130:57285)
0924 16:02:29.264093 (+ 12897us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.264172 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.266180 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.267078 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.267087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.267559 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:29.268186 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.268205 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.269450 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.269456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.271330 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.271346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.272647 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.272655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.272949 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.273737 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:29.273761 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:29.276117 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:29.279439 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:29.279448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:29.279468 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:29.279914 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:29.280314 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:29.280319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:29.280323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:29.280545 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:29.281104 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:29.281111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:29.281466 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.282407 (+ 941us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.283295 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12497,"thread_start_us":12269,"threads_started":1}
I20260924 16:02:29.286103 30752 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45089
I20260924 16:02:29.286492 30752 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:29.287290 30752 heartbeater.cc:511] Master 127.16.233.190:45089 requested a full tablet report, sending...
I20260924 16:02:29.289034 30401 ts_manager.cc:194] Registered new tserver with Master: ab2fe64f33524f7e933b12621e372070 (127.16.233.130:41175)
I20260924 16:02:29.290802 30401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:57285
I20260924 16:02:29.613166 30605 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:29.613606 30605 heartbeater.cc:511] Master 127.16.233.190:45089 requested a full tablet report, sending...
I20260924 16:02:29.615118 30401 ts_manager.cc:194] Registered new tserver with Master: 2ee054632d45464fa24c5759f4e489ea (127.16.233.129:46551)
I20260924 16:02:29.616425 30401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:54827
Sep 24 16:02:29 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
W20260924 16:02:29.712529 30756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:29.712963 30756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:29.713152 30756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:29.725222 30756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:29.725405 30756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:29.725677 30756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:02:29.738749 30756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.233.148:58409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:36241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=36049
--tserver_master_addrs=127.16.233.190:45089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:29.740217 30756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:29.741905 30756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:29.757071 30761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:29.757035 30762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:29.757280 30764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:29.759742 30756 server_base.cc:1034] running on GCE node
I20260924 16:02:29.760514 30756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:29.761641 30756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:29.762876 30756 hybrid_clock.cc:648] HybridClock initialized: now 1790265749762800 us; error 67 us; skew 500 ppm
I20260924 16:02:29.763283 30756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:29 dist-test-slave-xn4h krb5kdc[29372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:29.766841 30756 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:02:29.768610 30756 webserver.cc:460] Webserver started at http://127.16.233.131:36049/ using document root <none> and password file <none>
I20260924 16:02:29.769366 30756 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:29.769527 30756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:02:29 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:29.794453 30756 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:02:29.805913 30772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:29.808182 30756 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.007s sys 0.003s
I20260924 16:02:29.808421 30756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "d82a060cce054dc390c3755b49d9fe8e"
format_stamp: "Formatted at 2026-09-24 16:02:26 on dist-test-slave-xn4h"
server_key: "d85acdcd3396c0550320e5700f635b81"
server_key_iv: "3cef83873977dc743bb1b4faedb055e3"
server_key_version: "kuduclusterkey@0"
I20260924 16:02:29.809994 30756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:29.857411 30756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:29.863196 30756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:29.863612 30756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:29.867923 30756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:29.870088 30756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:29.870296 30756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:02:29.870517 30756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:29.870646 30756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:29.925197 30756 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:36241
I20260924 16:02:29.925282 30885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:36241 every 8 connection(s)
I20260924 16:02:29.927325 30756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:29.929242 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 30756
Sep 24 16:02:29 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:29.972110 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.930425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.16.233.190:45089)
0924 16:02:29.947652 (+ 17227us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.947659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.947682 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:02:29.947768 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.947774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.947862 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.951847 (+ 3985us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:29.952033 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.953184 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.956906 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.957623 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.963009 (+ 5386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.963420 (+ 411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:29.963427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:29.963432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:29.963475 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:29.967772 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.968786 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.968792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.968796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.968944 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.969432 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.969436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.969440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.969804 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:29.971330 (+ 1526us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.971469 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.971713 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17083,"spinlock_wait_cycles":59392,"thread_start_us":16937,"threads_started":1}
I20260924 16:02:29.973546 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.938532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45089 (local address 127.16.233.131:43621)
0924 16:02:29.942143 (+ 3611us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.942205 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.945553 (+ 3348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.951148 (+ 5595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.951163 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.951921 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:29.952872 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.952904 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.954815 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.954821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.957000 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.957024 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.958354 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.958365 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.958708 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.959684 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:29.959718 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:29.962800 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:29.967552 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:29.967565 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:29.967586 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:29.968305 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:29.968813 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:29.968818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:29.968822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:29.969139 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:29.969886 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:29.969896 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:29.970443 (+ 547us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.970781 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.971410 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3279,"thread_start_us":3093,"threads_started":1}
I20260924 16:02:29.976404 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.937904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:43621 (local address 127.16.233.190:45089)
0924 16:02:29.950240 (+ 12336us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.950247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.950269 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:02:29.950365 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.950371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.950465 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.953114 (+ 2649us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:29.953386 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.954590 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.957292 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.957991 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.959299 (+ 1308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.963155 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:29.963162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:29.963167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:29.963240 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:29.967327 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.968501 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.968507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.968511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.968672 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.969311 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.969316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.969320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.969726 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:29.975694 (+ 5968us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.975860 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.976104 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":12195,"spinlock_wait_cycles":2304,"thread_start_us":12683,"threads_started":1}
I20260924 16:02:29.977623 30886 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45089
I20260924 16:02:29.978022 30886 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:29.978920 30886 heartbeater.cc:511] Master 127.16.233.190:45089 requested a full tablet report, sending...
I20260924 16:02:29.980682 30401 ts_manager.cc:194] Registered new tserver with Master: d82a060cce054dc390c3755b49d9fe8e (127.16.233.131:36241)
I20260924 16:02:29.981564 30401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:43621
I20260924 16:02:29.988493 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:02:29 dist-test-slave-xn4h krb5kdc[29372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for HTTP/127.16.233.148@KRBTEST.COM
I20260924 16:02:30.027700 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.008789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.16.233.190:45089)
0924 16:02:30.009030 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.009039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.009654 (+ 615us) server_negotiation.cc:407] Connection header received
0924 16:02:30.009879 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.009885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.009984 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.010097 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:30.012226 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.013345 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.015577 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.016315 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.016428 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.020237 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:30.020244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:30.020249 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:30.020299 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:30.024039 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.025040 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.025046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.025050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.025228 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.025768 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.025773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.025777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.026200 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:30.026473 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.027094 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.027347 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:02:30.035368 30401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:30.047212 30401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:30.154261 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.095050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41175 (local address 127.0.0.1:47242)
0924 16:02:30.123874 (+ 28824us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.123892 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.124035 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.129230 (+ 5195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.129239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.129276 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:30.129727 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.129745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.134268 (+ 4523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.134274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.150780 (+ 16506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.150794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.153183 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.153190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.153427 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.153494 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.153636 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.153861 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28699,"thread_start_us":26612,"threads_started":1}
I20260924 16:02:30.155246 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.095632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.16.233.130:41175)
0924 16:02:30.121448 (+ 25816us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.121458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.127339 (+ 5881us) server_negotiation.cc:407] Connection header received
0924 16:02:30.127482 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.127504 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.127818 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.128064 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:30.130566 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.132174 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.150985 (+ 18811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.152600 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.152759 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.152953 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.154712 (+ 1759us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.154923 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25638,"thread_start_us":25547,"threads_started":1}
I20260924 16:02:30.156104 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.102156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.16.233.129:46551)
0924 16:02:30.134039 (+ 31883us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.134049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.142192 (+ 8143us) server_negotiation.cc:407] Connection header received
0924 16:02:30.142343 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.142352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.142690 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.144514 (+ 1824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:30.145463 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.148581 (+ 3118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.150818 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.153827 (+ 3009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.154024 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.154193 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.155536 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.155777 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31656,"thread_start_us":31900,"threads_started":1}
I20260924 16:02:30.160475 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.084072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.16.233.131:36241)
0924 16:02:30.121009 (+ 36937us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.121021 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.128317 (+ 7296us) server_negotiation.cc:407] Connection header received
0924 16:02:30.128428 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.128435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.128700 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.128885 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:30.132401 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.134090 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.155804 (+ 21714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.157603 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.157784 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.157979 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.159904 (+ 1925us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.160123 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36709,"thread_start_us":38614,"threads_started":1}
I20260924 16:02:30.155339 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.101601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.0.0.1:42286)
0924 16:02:30.141854 (+ 40253us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.141883 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.142040 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.144848 (+ 2808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.144854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.144887 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:30.145239 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.145252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.148785 (+ 3533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.148791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.150584 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.150603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.154390 (+ 3787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.154398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.154709 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.154747 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.154836 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.154971 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40112,"thread_start_us":39507,"threads_started":1}
I20260924 16:02:30.163556 30684 tablet_service.cc:1467] Processing CreateTablet for tablet e86aa63605ce4cffaf838cc6f783aad0 (DEFAULT_TABLE table=test-table [id=666be70bef464db0b127b8c43c0e6bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:30.165050 30684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86aa63605ce4cffaf838cc6f783aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:30.166615 30538 tablet_service.cc:1467] Processing CreateTablet for tablet e86aa63605ce4cffaf838cc6f783aad0 (DEFAULT_TABLE table=test-table [id=666be70bef464db0b127b8c43c0e6bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:30.168126 30538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86aa63605ce4cffaf838cc6f783aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:30.159132 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.083361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:36241 (local address 127.0.0.1:58250)
0924 16:02:30.122106 (+ 38745us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.122170 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.122444 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.129929 (+ 7485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.129934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.129962 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:30.130327 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.130341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.134375 (+ 4034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.134380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.139292 (+ 4912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.139321 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.158144 (+ 18823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.158149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.158277 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.158348 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.158435 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.158622 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38441,"mutex_wait_us":1,"thread_start_us":40833,"threads_started":1}
I20260924 16:02:30.179029 30820 tablet_service.cc:1467] Processing CreateTablet for tablet e86aa63605ce4cffaf838cc6f783aad0 (DEFAULT_TABLE table=test-table [id=666be70bef464db0b127b8c43c0e6bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:30.180644 30820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86aa63605ce4cffaf838cc6f783aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:30.187120 30909 tablet_bootstrap.cc:492] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: Bootstrap starting.
I20260924 16:02:30.192674 30910 tablet_bootstrap.cc:492] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea: Bootstrap starting.
I20260924 16:02:30.193122 30909 tablet_bootstrap.cc:654] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:30.194239 30909 log.cc:824] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:30.196415 30910 tablet_bootstrap.cc:654] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:30.199536 30910 log.cc:824] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:30.203999 30909 tablet_bootstrap.cc:492] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: No bootstrap required, opened a new log
I20260924 16:02:30.204411 30909 ts_tablet_manager.cc:1397] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260924 16:02:30.216888 30909 raft_consensus.cc:348] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } }
I20260924 16:02:30.218101 30910 tablet_bootstrap.cc:492] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea: No bootstrap required, opened a new log
I20260924 16:02:30.218204 30913 tablet_bootstrap.cc:492] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: Bootstrap starting.
I20260924 16:02:30.218587 30910 ts_tablet_manager.cc:1397] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.014s
I20260924 16:02:30.222450 30909 raft_consensus.cc:374] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:30.222668 30909 raft_consensus.cc:729] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab2fe64f33524f7e933b12621e372070, State: Initialized, Role: FOLLOWER
I20260924 16:02:30.223645 30909 consensus_queue.cc:260] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } }
I20260924 16:02:30.222290 30913 tablet_bootstrap.cc:654] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:30.225982 30913 log.cc:824] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:30.229615 30752 heartbeater.cc:503] Master 127.16.233.190:45089 was elected leader, sending a full tablet report...
I20260924 16:02:30.230798 30909 ts_tablet_manager.cc:1428] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: Time spent starting tablet: real 0.026s user 0.010s sys 0.006s
I20260924 16:02:30.231062 30913 tablet_bootstrap.cc:492] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: No bootstrap required, opened a new log
I20260924 16:02:30.231405 30913 ts_tablet_manager.cc:1397] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260924 16:02:30.237893 30910 raft_consensus.cc:348] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } }
I20260924 16:02:30.238631 30910 raft_consensus.cc:374] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:30.238794 30910 raft_consensus.cc:729] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee054632d45464fa24c5759f4e489ea, State: Initialized, Role: FOLLOWER
I20260924 16:02:30.239748 30910 consensus_queue.cc:260] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } }
I20260924 16:02:30.242348 30913 raft_consensus.cc:348] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } }
I20260924 16:02:30.244061 30913 raft_consensus.cc:374] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:30.244238 30913 raft_consensus.cc:729] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82a060cce054dc390c3755b49d9fe8e, State: Initialized, Role: FOLLOWER
I20260924 16:02:30.245033 30913 consensus_queue.cc:260] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } }
W20260924 16:02:30.264078 30753 tablet.cc:2369] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:30.264161 30605 heartbeater.cc:503] Master 127.16.233.190:45089 was elected leader, sending a full tablet report...
I20260924 16:02:30.264915 30910 ts_tablet_manager.cc:1428] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea: Time spent starting tablet: real 0.046s user 0.008s sys 0.026s
I20260924 16:02:30.285105 30886 heartbeater.cc:503] Master 127.16.233.190:45089 was elected leader, sending a full tablet report...
I20260924 16:02:30.289206 30918 raft_consensus.cc:482] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:30.289649 30918 raft_consensus.cc:504] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } }
I20260924 16:02:30.291255 30913 ts_tablet_manager.cc:1428] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: Time spent starting tablet: real 0.060s user 0.029s sys 0.027s
I20260924 16:02:30.294802 30918 leader_election.cc:288] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab2fe64f33524f7e933b12621e372070 (127.16.233.130:41175), 2ee054632d45464fa24c5759f4e489ea (127.16.233.129:46551)
I20260924 16:02:30.302623 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.292119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41175 (local address 127.16.233.131:43729)
0924 16:02:30.292311 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.292333 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.292450 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.293372 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.293377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.293406 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:30.293601 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.293613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.295035 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.295042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.297531 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.297557 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.301794 (+ 4237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.301800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.301994 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.302030 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.302142 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.302297 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:02:30.303427 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.292729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:43729 (local address 127.16.233.130:41175)
0924 16:02:30.292927 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.292933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.292953 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:02:30.293022 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.293027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.293097 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.293191 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:30.293776 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.294860 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.299666 (+ 4806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.301426 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.301604 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.301677 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.303009 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.303164 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:02:30.305441 30704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86aa63605ce4cffaf838cc6f783aad0" candidate_uuid: "d82a060cce054dc390c3755b49d9fe8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab2fe64f33524f7e933b12621e372070" is_pre_election: true
I20260924 16:02:30.306103 30704 raft_consensus.cc:2454] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d82a060cce054dc390c3755b49d9fe8e in term 0.
I20260924 16:02:30.307142 30774 leader_election.cc:302] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab2fe64f33524f7e933b12621e372070, d82a060cce054dc390c3755b49d9fe8e; no voters:
I20260924 16:02:30.307775 30918 raft_consensus.cc:2781] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:30.307996 30918 raft_consensus.cc:482] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:30.308100 30918 raft_consensus.cc:3037] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 0 FOLLOWER]: Advancing to term 1
W20260924 16:02:30.310249 30606 tablet.cc:2369] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:30.311424 30918 raft_consensus.cc:504] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } }
I20260924 16:02:30.312258 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.295924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:49795 (local address 127.16.233.129:46551)
0924 16:02:30.296150 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.296157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.304110 (+ 7953us) server_negotiation.cc:407] Connection header received
0924 16:02:30.305034 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.305040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.305129 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.305197 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:30.305841 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.306952 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.309281 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.310672 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.310807 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.310887 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.311704 (+ 817us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.311906 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:02:30.313097 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.295293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.16.233.131:49795)
0924 16:02:30.304009 (+ 8716us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.304029 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.304230 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.305416 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.305421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.305451 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:30.305664 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.305676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.307153 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.307158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.309074 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.309092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.311079 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.311088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.311401 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.311478 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.312456 (+ 978us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.312687 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8603,"threads_started":1}
I20260924 16:02:30.313007 30704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86aa63605ce4cffaf838cc6f783aad0" candidate_uuid: "d82a060cce054dc390c3755b49d9fe8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab2fe64f33524f7e933b12621e372070"
I20260924 16:02:30.313238 30704 raft_consensus.cc:3037] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:30.314569 30558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86aa63605ce4cffaf838cc6f783aad0" candidate_uuid: "d82a060cce054dc390c3755b49d9fe8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee054632d45464fa24c5759f4e489ea" is_pre_election: true
I20260924 16:02:30.314989 30558 raft_consensus.cc:2454] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d82a060cce054dc390c3755b49d9fe8e in term 0.
I20260924 16:02:30.316401 30704 raft_consensus.cc:2454] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d82a060cce054dc390c3755b49d9fe8e in term 1.
I20260924 16:02:30.316988 30774 leader_election.cc:302] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab2fe64f33524f7e933b12621e372070, d82a060cce054dc390c3755b49d9fe8e; no voters:
I20260924 16:02:30.317296 30917 raft_consensus.cc:2781] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:30.317865 30558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86aa63605ce4cffaf838cc6f783aad0" candidate_uuid: "d82a060cce054dc390c3755b49d9fe8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee054632d45464fa24c5759f4e489ea"
I20260924 16:02:30.318099 30558 raft_consensus.cc:3037] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:30.321425 30558 raft_consensus.cc:2454] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d82a060cce054dc390c3755b49d9fe8e in term 1.
I20260924 16:02:30.322130 30918 leader_election.cc:288] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [CANDIDATE]: Term 1 election: Requested vote from peers ab2fe64f33524f7e933b12621e372070 (127.16.233.130:41175), 2ee054632d45464fa24c5759f4e489ea (127.16.233.129:46551)
I20260924 16:02:30.324400 30917 raft_consensus.cc:686] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 1 LEADER]: Becoming Leader. State: Replica: d82a060cce054dc390c3755b49d9fe8e, State: Running, Role: LEADER
I20260924 16:02:30.325089 30917 consensus_queue.cc:237] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } }
I20260924 16:02:30.330832 30401 catalog_manager.cc:5495] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e reported cstate change: term changed from 0 to 1, leader changed from <none> to d82a060cce054dc390c3755b49d9fe8e (127.16.233.131). New cstate: current_term: 1 leader_uuid: "d82a060cce054dc390c3755b49d9fe8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82a060cce054dc390c3755b49d9fe8e" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:30.354876 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.346722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58254 (local address 127.16.233.131:36241)
0924 16:02:30.346947 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.346953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.346975 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:02:30.347094 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.347100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.347223 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.347389 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:30.348053 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.349200 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.351361 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.352001 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.352131 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.352910 (+ 779us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.353663 (+ 753us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.353808 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.354508 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.354598 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:02:30.369050 30558 raft_consensus.cc:1261] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [term 1 FOLLOWER]: Refusing update from remote peer d82a060cce054dc390c3755b49d9fe8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:30.369235 30704 raft_consensus.cc:1261] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [term 1 FOLLOWER]: Refusing update from remote peer d82a060cce054dc390c3755b49d9fe8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:30.370508 30917 consensus_queue.cc:1035] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ee054632d45464fa24c5759f4e489ea" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.370957 30920 consensus_queue.cc:1035] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab2fe64f33524f7e933b12621e372070" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.389396 30925 mvcc.cc:205] Tried to move back new op lower bound from 7332928513496297472 to 7332928513336905728. Current Snapshot: MvccSnapshot[applied={T|T < 7332928513496297472}]
I20260924 16:02:30.396198 30926 mvcc.cc:205] Tried to move back new op lower bound from 7332928513496297472 to 7332928513336905728. Current Snapshot: MvccSnapshot[applied={T|T < 7332928513496297472}]
I20260924 16:02:30.423658 30401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:30.423950 30401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:30.427736 30401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 226905715b864d858142ac073514f6c7: Sending DeleteTablet for 3 replicas of tablet e86aa63605ce4cffaf838cc6f783aad0
I20260924 16:02:30.429035 30820 tablet_service.cc:1514] Processing DeleteTablet for tablet e86aa63605ce4cffaf838cc6f783aad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:58250
I20260924 16:02:30.429127 30684 tablet_service.cc:1514] Processing DeleteTablet for tablet e86aa63605ce4cffaf838cc6f783aad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:47242
I20260924 16:02:30.430389 30538 tablet_service.cc:1514] Processing DeleteTablet for tablet e86aa63605ce4cffaf838cc6f783aad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:42286
I20260924 16:02:30.439034 30941 tablet_replica.cc:330] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea: stopping tablet replica
I20260924 16:02:30.439817 30941 raft_consensus.cc:2229] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:30.440713 30941 raft_consensus.cc:2258] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.442734 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 30439
I20260924 16:02:30.445010 30941 ts_tablet_manager.cc:1905] T e86aa63605ce4cffaf838cc6f783aad0 P 2ee054632d45464fa24c5759f4e489ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.445866 30939 tablet_replica.cc:330] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: stopping tablet replica
I20260924 16:02:30.446560 30940 tablet_replica.cc:330] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: stopping tablet replica
I20260924 16:02:30.447095 30940 raft_consensus.cc:2229] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:30.447634 30939 raft_consensus.cc:2229] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:30.448354 30939 raft_consensus.cc:2258] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.449124 30940 raft_consensus.cc:2258] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.451570 30940 ts_tablet_manager.cc:1905] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.453752 30939 ts_tablet_manager.cc:1905] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.460943 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.445261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.16.233.129:46551)
0924 16:02:30.445496 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.445502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.445523 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:02:30.445589 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.445593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.445654 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.445772 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:30.447737 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.448530 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.450288 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.450731 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.450872 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.454250 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:30.454286 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:30.454306 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:30.454371 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:30.457579 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.458498 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.458508 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.458512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.458669 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.459078 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.459086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.459089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.459563 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:30.459905 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.460205 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.460450 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 16:02:30.461366 30940 ts_tablet_manager.cc:1918] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:30.461680 30600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:30.462615 30940 log.cc:1196] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/e86aa63605ce4cffaf838cc6f783aad0
I20260924 16:02:30.463600 30940 ts_tablet_manager.cc:1939] T e86aa63605ce4cffaf838cc6f783aad0 P ab2fe64f33524f7e933b12621e372070: Deleting consensus metadata
I20260924 16:02:30.466612 30386 catalog_manager.cc:4841] TS ab2fe64f33524f7e933b12621e372070 (127.16.233.130:41175): tablet e86aa63605ce4cffaf838cc6f783aad0 (table test-table [id=666be70bef464db0b127b8c43c0e6bb3]) successfully deleted
I20260924 16:02:30.469547 30939 ts_tablet_manager.cc:1918] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:30.469863 30939 log.cc:1196] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/e86aa63605ce4cffaf838cc6f783aad0
I20260924 16:02:30.470659 30939 ts_tablet_manager.cc:1939] T e86aa63605ce4cffaf838cc6f783aad0 P d82a060cce054dc390c3755b49d9fe8e: Deleting consensus metadata
I20260924 16:02:30.472646 30385 catalog_manager.cc:4841] TS d82a060cce054dc390c3755b49d9fe8e (127.16.233.131:36241): tablet e86aa63605ce4cffaf838cc6f783aad0 (table test-table [id=666be70bef464db0b127b8c43c0e6bb3]) successfully deleted
I20260924 16:02:30.560555 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 30439
W20260924 16:02:30.585170 30387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.129:46551 (error 108)
I20260924 16:02:30.585944 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 30609
I20260924 16:02:30.587357 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.586754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.0.0.1:42304)
0924 16:02:30.587029 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.587162 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260924 16:02:30.587656 30387 catalog_manager.cc:4584] TS 2ee054632d45464fa24c5759f4e489ea (127.16.233.129:46551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e86aa63605ce4cffaf838cc6f783aad0: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46551: connect: Connection refused (error 111)
I20260924 16:02:30.602154 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.586851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.16.233.130:41175)
0924 16:02:30.587035 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.587042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.587110 (+ 68us) server_negotiation.cc:407] Connection header received
0924 16:02:30.587344 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.587350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.587445 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.587570 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:30.589030 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.590170 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.591612 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.592267 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.592407 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.595055 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:30.595076 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:30.595098 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:30.595146 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:30.599040 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.599771 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.599781 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.599784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.599969 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.600401 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.600406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.600409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.600927 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:30.601224 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.601501 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.601773 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:02:30.602933 30745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:30.641513 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.640884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.0.0.1:42320)
0924 16:02:30.641165 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.641310 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 16:02:30.642995 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.642350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.0.0.1:42322)
0924 16:02:30.642584 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.642850 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260924 16:02:30.696246 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 30609
I20260924 16:02:30.703603 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.703155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.0.0.1:42334)
0924 16:02:30.703364 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.703451 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:02:30.704658 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.704332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.0.0.1:42342)
0924 16:02:30.704477 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.704552 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 16:02:30.719666 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 30756
I20260924 16:02:30.736511 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.720498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58264 (local address 127.16.233.131:36241)
0924 16:02:30.720671 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.720676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.720770 (+ 94us) server_negotiation.cc:407] Connection header received
0924 16:02:30.720945 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.720949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.721012 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.721109 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:30.723061 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.723888 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.725851 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.726521 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.726628 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.730054 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:30.730067 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:30.730082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:30.730127 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:30.733239 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.734161 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.734168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.734170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.734365 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.735012 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.735016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.735018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.735383 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:30.735659 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.736011 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.736205 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:02:30.737228 30881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:30.785274 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.784784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.0.0.1:42356)
0924 16:02:30.784988 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.785079 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 16:02:30.786290 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.785907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.0.0.1:42370)
0924 16:02:30.786104 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.786179 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 16:02:30.860888 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 30756
I20260924 16:02:30.880275 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 30368
I20260924 16:02:30.881500 30432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:30.997313 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.996721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46551 (local address 127.0.0.1:42378)
0924 16:02:30.996947 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.997066 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:02:30.997458 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 30368
I20260924 16:02:31.020519 17318 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:02:31.356485 17318 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:02:31.356732 17318 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:02:31.702900 17318 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:02:31.703 UTC [29403] LOG: received smart shutdown request
2026-09-24 16:02:31.707 UTC [29403] LOG: background worker "logical replication launcher" (PID 29412) exited with exit code 1
2026-09-24 16:02:31.707 UTC [29407] LOG: shutting down
2026-09-24 16:02:31.872 UTC [29403] LOG: database system is shut down
2026-09-24T16:02:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53541 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:32 dist-test-slave-xn4h krb5kdc[30955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:32 dist-test-slave-xn4h krb5kdc[30955](info): set up 2 sockets
Sep 24 16:02:32 dist-test-slave-xn4h krb5kdc[30955](info): commencing operation
krb5kdc: starting...
W20260924 16:02:33.112128 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:02:33 dist-test-slave-xn4h krb5kdc[30955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:02:33.354673 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34359
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:34359,127.16.233.189:41715,127.16.233.188:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:33.701958 30969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:33.702275 30969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:33.702478 30969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:33.709916 30969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:33.710047 30969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:33.710125 30969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:33.710210 30969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:33.710302 30969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:33.721271 30969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:34359,127.16.233.189:41715,127.16.233.188:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:33.722859 30969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:33.724329 30969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:33.736840 30974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:33.736891 30975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:33.736958 30977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:33.738039 30969 server_base.cc:1034] running on GCE node
I20260924 16:02:33.738989 30969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:33.740537 30969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:33.741786 30969 hybrid_clock.cc:648] HybridClock initialized: now 1790265753741741 us; error 40 us; skew 500 ppm
I20260924 16:02:33.742210 30969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:33 dist-test-slave-xn4h krb5kdc[30955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:33.756783 30969 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:02:33.758703 30969 webserver.cc:460] Webserver started at http://127.16.233.190:41373/ using document root <none> and password file <none>
I20260924 16:02:33.759500 30969 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:33.759665 30969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:33.760010 30969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:33.763338 30969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "71e36eacd995449a876bbd97c570c2df"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-xn4h"
server_key: "61b26dccc1d2d31dae51921d6cf2b426"
server_key_iv: "f08b30cc51ab4321107001491b6d1585"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.764245 30969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "71e36eacd995449a876bbd97c570c2df"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-xn4h"
server_key: "61b26dccc1d2d31dae51921d6cf2b426"
server_key_iv: "f08b30cc51ab4321107001491b6d1585"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.770084 30969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 16:02:33.774214 30984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:33.775846 30969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:33.776073 30969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "71e36eacd995449a876bbd97c570c2df"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-xn4h"
server_key: "61b26dccc1d2d31dae51921d6cf2b426"
server_key_iv: "f08b30cc51ab4321107001491b6d1585"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.776302 30969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:33.807022 30969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:33.811342 30969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:33.811662 30969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:33.830802 30969 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:34359
I20260924 16:02:33.830827 31037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:34359 every 8 connection(s)
I20260924 16:02:33.832252 30969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:33.836014 31038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:33.838798 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 30969
I20260924 16:02:33.839041 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:02:33.839843 17318 external_mini_cluster.cc:1428] Setting key 4b9847e6ebf8f937847bb83746d89e0c
I20260924 16:02:33.845762 31038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:02:33 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.189@KRBTEST.COM" created.
I20260924 16:02:33.873217 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.850146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.16.233.190:34359)
0924 16:02:33.852272 (+ 2126us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:33.852281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:33.852322 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:02:33.854785 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:33.854791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:33.855062 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:33.855237 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:33.857101 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.858372 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.860795 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.861366 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.861505 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:33.865330 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:33.865345 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:33.865369 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:33.865415 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:33.868607 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:33.869582 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:33.869589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:33.869591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:33.869740 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:33.870225 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:33.870228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:33.870230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:33.870854 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:33.871022 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:33.871756 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 16:02:33.872087 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":1950,"spinlock_wait_cycles":121216,"thread_start_us":1707,"threads_started":1}
I20260924 16:02:33.873273 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.850146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:34359 (local address 127.0.0.1:45936)
0924 16:02:33.851229 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:33.851286 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:33.854355 (+ 3069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:33.855332 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:33.855342 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:33.856245 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:33.856789 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:33.856820 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.858553 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.858562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:33.860626 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:33.860646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.861624 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.861632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:33.861800 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:33.862550 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:33.862573 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:33.865038 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:33.868762 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:33.868779 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:33.868796 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:33.869442 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:33.869856 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:33.869864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:33.869870 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:33.870106 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:33.871018 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:33.871030 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:33.871332 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 16:02:33.871630 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 16:02:33.872293 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":59,"thread_start_us":542,"threads_started":1}
I20260924 16:02:33.876459 31038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:02:33.877748 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.877212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58306)
0924 16:02:33.877424 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:33.877579 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 16:02:33.878059 30987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
I20260924 16:02:33.879938 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.879432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58312)
0924 16:02:33.879632 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:33.879732 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:02:33.880206 31038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:41715: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab.
Entry for principal kudu/127.16.233.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab.
I20260924 16:02:33.897598 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.189:41715
--webserver_interface=127.16.233.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab
--principal=kudu/127.16.233.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:34359,127.16.233.189:41715,127.16.233.188:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:33.929585 31038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } attempt: 1
I20260924 16:02:33.931057 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.930501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58316)
0924 16:02:33.930723 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:33.930901 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:02:33.932111 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.931760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58318)
0924 16:02:33.931913 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:33.931977 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:02:33.932317 31038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:41715: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
I20260924 16:02:34.000618 31038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } attempt: 2
I20260924 16:02:34.001838 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.001350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58322)
0924 16:02:34.001607 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.001699 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:02:34.002943 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.002569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58332)
0924 16:02:34.002730 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.002805 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 16:02:34.003150 31038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:41715: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
I20260924 16:02:34.094437 31038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } attempt: 3
I20260924 16:02:34.095636 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.095176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58338)
0924 16:02:34.095381 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.095498 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 16:02:34.096784 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.096416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58346)
0924 16:02:34.096572 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.096641 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:02:34.097064 31038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:41715: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
I20260924 16:02:34.240358 31038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } attempt: 4
I20260924 16:02:34.241667 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.241074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58358)
0924 16:02:34.241435 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.241528 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
W20260924 16:02:34.241806 31045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:34.242238 31045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:34.242447 31045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 16:02:34.242882 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.242505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58368)
0924 16:02:34.242660 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.242755 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 16:02:34.243114 31038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:41715: Network error: Client connection negotiation failed: client connection to 127.16.233.189:41715: connect: Connection refused (error 111)
W20260924 16:02:34.250398 31045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:34.250546 31045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:34.250650 31045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:34.250742 31045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:34.250808 31045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:34.261642 31045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:34359,127.16.233.189:41715,127.16.233.188:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab
--principal=kudu/127.16.233.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.189:41715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.233.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:34.263134 31045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:34.264506 31045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:34.276834 31051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.276894 31050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.276894 31053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:34.278149 31045 server_base.cc:1034] running on GCE node
I20260924 16:02:34.279030 31045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:34.280758 31045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:34.281991 31045 hybrid_clock.cc:648] HybridClock initialized: now 1790265754281937 us; error 56 us; skew 500 ppm
I20260924 16:02:34.282382 31045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:34.285717 31045 init.cc:377] Logged in from keytab as kudu/127.16.233.189@KRBTEST.COM (short username kudu)
I20260924 16:02:34.287386 31045 webserver.cc:460] Webserver started at http://127.16.233.189:42663/ using document root <none> and password file <none>
I20260924 16:02:34.288089 31045 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:34.288208 31045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:34.288476 31045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:34.291675 31045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/data/instance:
uuid: "e65efdbd110a4c51984121619c4dfe14"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-xn4h"
server_key: "abd4607f58b314c886a06fc383476994"
server_key_iv: "f442154a17fdf18518d393881915c1e1"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.292486 31045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/wal/instance:
uuid: "e65efdbd110a4c51984121619c4dfe14"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-xn4h"
server_key: "abd4607f58b314c886a06fc383476994"
server_key_iv: "f442154a17fdf18518d393881915c1e1"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.298004 31045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:02:34.301661 31060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.303169 31045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:34.303360 31045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/wal
uuid: "e65efdbd110a4c51984121619c4dfe14"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-xn4h"
server_key: "abd4607f58b314c886a06fc383476994"
server_key_iv: "f442154a17fdf18518d393881915c1e1"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.303572 31045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:34.330225 31045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:34.335290 31045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:34.335618 31045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:34.355762 31045 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.189:41715
I20260924 16:02:34.355792 31112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.189:41715 every 8 connection(s)
I20260924 16:02:34.357659 31045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
I20260924 16:02:34.361349 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 31045
I20260924 16:02:34.361577 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-1/wal/instance
I20260924 16:02:34.361910 31113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:34.362129 17318 external_mini_cluster.cc:1428] Setting key 81fe4a5572993ee2ac8a45e9a96d43be
I20260924 16:02:34.369073 31113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.233.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.188@KRBTEST.COM" created.
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:34.395431 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.372065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.16.233.190:34359)
0924 16:02:34.372263 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.372268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.373931 (+ 1663us) server_negotiation.cc:407] Connection header received
0924 16:02:34.376020 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.376027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.376129 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.376250 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.377879 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.378646 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.380369 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.380848 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.380991 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.388273 (+ 7282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.388279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.388283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.388325 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.391364 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.392575 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.392579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.392582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.392718 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.393322 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.393332 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.393334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.393688 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:34.393862 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:34.394808 (+ 946us) server_negotiation.cc:299] Negotiation successful
0924 16:02:34.395070 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:02:34.396589 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.372850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:34359 (local address 127.0.0.1:45940)
0924 16:02:34.373743 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.373790 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.375708 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.376506 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.376515 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.377065 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.377683 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.377703 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.378821 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.378826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.380188 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.380202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.381089 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.381097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.381374 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.382203 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.382227 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.387905 (+ 5678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:34.391576 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.391592 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.391616 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.392424 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.392847 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.392855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.392861 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.393163 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.393838 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:34.393848 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:34.394359 (+ 511us) client_negotiation.cc:769] Sending connection context
0924 16:02:34.394699 (+ 340us) client_negotiation.cc:240] Negotiation successful
0924 16:02:34.395291 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":380,"threads_started":1}
I20260924 16:02:34.398559 31113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:02:34.415148 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.399444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58374)
0924 16:02:34.399611 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.399627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.399734 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.400942 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.400950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.401481 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.401721 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.401738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.403190 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.403198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.404811 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.404830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.405692 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.405706 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.405987 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.406693 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.406718 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.409284 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:34.412556 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.412559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.412562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.412962 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.413346 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.413349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.413351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.413488 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.414190 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:34.414193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:34.414300 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 16:02:34.414487 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 16:02:34.414709 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 16:02:34.415163 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.399790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.16.233.189:41715)
0924 16:02:34.400479 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.400485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.400512 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:02:34.400565 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.400571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.400767 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.400887 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.401890 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.403036 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.404965 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.405526 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.405625 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.409487 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.409500 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.409514 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.409543 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.412436 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.413064 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.413070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.413072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.413249 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.413581 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.413584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.413586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.414042 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:34.414236 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:34.414612 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 16:02:34.414799 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":407,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab.
Entry for principal kudu/127.16.233.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab.
I20260924 16:02:34.418046 31113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:02:34.418375 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.188:44921
--webserver_interface=127.16.233.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab
--principal=kudu/127.16.233.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:34359,127.16.233.189:41715,127.16.233.188:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:34.419579 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.418942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40194)
0924 16:02:34.419158 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.419350 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 16:02:34.419919 31062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
I20260924 16:02:34.421789 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.421256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40210)
0924 16:02:34.421481 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.421599 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 16:02:34.422124 31113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:44921: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
I20260924 16:02:34.471570 31113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } attempt: 1
I20260924 16:02:34.472864 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.472390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40218)
0924 16:02:34.472592 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.472708 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 16:02:34.473976 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.473638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40224)
0924 16:02:34.473788 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.473862 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:02:34.474288 31113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:44921: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
I20260924 16:02:34.542402 31038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } attempt: 5
I20260924 16:02:34.542520 31113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } attempt: 2
I20260924 16:02:34.544431 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.543914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40236)
0924 16:02:34.544116 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.544225 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 16:02:34.546628 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.546147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40240)
0924 16:02:34.546334 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.546431 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:02:34.546974 31113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:44921: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:02:34.557395 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.544151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.16.233.189:41715)
0924 16:02:34.544323 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.544333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.544349 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:02:34.544400 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.544404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.544490 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.544609 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.545505 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.546543 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.547901 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.548427 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.548566 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.551788 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.551796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.551802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.551849 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.554675 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.555404 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.555411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.555414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.555547 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.555944 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.555950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.555953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.556265 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:34.556429 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:34.556819 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 16:02:34.557078 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 16:02:34.557425 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.543145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58390)
0924 16:02:34.543402 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.543425 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.543563 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.544720 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.544725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.545134 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.545328 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.545338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.546691 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.546699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.547741 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.547757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.548585 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.548591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.548804 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.549444 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.549462 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.551543 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:34.554840 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.554846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.554851 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.555269 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.555666 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.555672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.555676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.555829 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.556387 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:34.556392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:34.556559 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 16:02:34.556716 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 16:02:34.557041 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 16:02:34.558548 31038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:02:34.559813 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.559249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40242)
0924 16:02:34.559452 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.559597 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 16:02:34.560894 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.560422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40244)
0924 16:02:34.560624 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.560737 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:02:34.561105 31038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:44921: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
I20260924 16:02:34.612381 31038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } attempt: 1
I20260924 16:02:34.613698 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.613113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40256)
0924 16:02:34.613355 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.613478 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 16:02:34.614842 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.614401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40264)
0924 16:02:34.614579 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.614677 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:02:34.615095 31038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:44921: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
I20260924 16:02:34.638254 31113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } attempt: 3
I20260924 16:02:34.639655 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.639071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40280)
0924 16:02:34.639312 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.639430 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 16:02:34.640971 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.640467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40286)
0924 16:02:34.640719 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.640817 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260924 16:02:34.641238 31113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:44921: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
I20260924 16:02:34.683338 31038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } attempt: 2
I20260924 16:02:34.684581 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.683994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40292)
0924 16:02:34.684229 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.684346 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 16:02:34.685734 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.685296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40298)
0924 16:02:34.685465 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.685566 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:02:34.685962 31038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:44921: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
W20260924 16:02:34.772193 31120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:34.772517 31120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:34.772691 31120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:34.780354 31120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:34.780494 31120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:34.780592 31120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:34.780691 31120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:34.780757 31120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:34.784626 31113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } attempt: 4
I20260924 16:02:34.785799 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.785315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40304)
0924 16:02:34.785546 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.785637 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 16:02:34.786877 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.786537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40310)
0924 16:02:34.786707 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.786774 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 16:02:34.787103 31113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:44921: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
I20260924 16:02:34.792181 31038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } attempt: 3
I20260924 16:02:34.793237 31120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:34359,127.16.233.189:41715,127.16.233.188:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab
--principal=kudu/127.16.233.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.188:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.233.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:34.793356 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.792845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40312)
0924 16:02:34.793111 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.793204 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 16:02:34.794654 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.794193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40328)
0924 16:02:34.794450 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.794531 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260924 16:02:34.794907 31038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:44921: Network error: Client connection negotiation failed: client connection to 127.16.233.188:44921: connect: Connection refused (error 111)
I20260924 16:02:34.794927 31120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:34.796270 31120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:34.808674 31128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.808801 31125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.808776 31126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:34.809376 31120 server_base.cc:1034] running on GCE node
I20260924 16:02:34.810189 31120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:34.811651 31120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:34.812871 31120 hybrid_clock.cc:648] HybridClock initialized: now 1790265754812819 us; error 45 us; skew 500 ppm
I20260924 16:02:34.813222 31120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:34.822760 31120 init.cc:377] Logged in from keytab as kudu/127.16.233.188@KRBTEST.COM (short username kudu)
I20260924 16:02:34.824473 31120 webserver.cc:460] Webserver started at http://127.16.233.188:35465/ using document root <none> and password file <none>
I20260924 16:02:34.825261 31120 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:34.825415 31120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:34.825735 31120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:34.828949 31120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/data/instance:
uuid: "61b7ca4449154fc0a41778e48a75376e"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-xn4h"
server_key: "3049856287317e6ff552d4443a8a4bca"
server_key_iv: "aaea81da38ed6bbbf0d7a3d75f0f4554"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.829824 31120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/wal/instance:
uuid: "61b7ca4449154fc0a41778e48a75376e"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-xn4h"
server_key: "3049856287317e6ff552d4443a8a4bca"
server_key_iv: "aaea81da38ed6bbbf0d7a3d75f0f4554"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.835752 31120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:02:34.839704 31135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.841279 31120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:34.841491 31120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/wal
uuid: "61b7ca4449154fc0a41778e48a75376e"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-xn4h"
server_key: "3049856287317e6ff552d4443a8a4bca"
server_key_iv: "aaea81da38ed6bbbf0d7a3d75f0f4554"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.841717 31120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:34.875696 31120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:34.882157 31120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:34.882521 31120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:34.901515 31120 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.188:44921
I20260924 16:02:34.901552 31186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.188:44921 every 8 connection(s)
I20260924 16:02:34.903100 31120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
I20260924 16:02:34.904011 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 31120
I20260924 16:02:34.904245 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265545293861-17318-0/minicluster-data/master-2/wal/instance
I20260924 16:02:34.904714 17318 external_mini_cluster.cc:1428] Setting key 1a63af48ad1b5445df78fe6e10a061e0
I20260924 16:02:34.907859 31187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:34.916604 31187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:34.932137 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.906726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.16.233.188:44921)
0924 16:02:34.907645 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.907657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.907699 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:02:34.909486 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.909508 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.909921 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.910292 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.912196 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.913521 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.915424 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.916079 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.916210 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.922181 (+ 5971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.922189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.922200 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.922241 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.927428 (+ 5187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.928245 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.928253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.928260 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.928429 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.928874 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.928878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.928880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.929471 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:34.929827 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:34.930269 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 16:02:34.930755 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":399,"threads_started":1}
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:34.955762 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.920090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:34359 (local address 127.0.0.1:45948)
0924 16:02:34.921094 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.921148 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.921354 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.921774 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.921781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.922552 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.922915 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.922931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.924559 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.924563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.926130 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.926152 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.927215 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.927220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.927342 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.928728 (+ 1386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.928751 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.931032 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:34.952503 (+ 21471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.952566 (+ 63us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.952577 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.953164 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.953623 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.953628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.953632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.953791 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.954474 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:34.954485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:34.954697 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 16:02:34.954888 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 16:02:34.955272 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":590,"threads_started":1}
I20260924 16:02:34.955773 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.920101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.16.233.190:34359)
0924 16:02:34.920742 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.920750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.921250 (+ 500us) server_negotiation.cc:407] Connection header received
0924 16:02:34.921485 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.921495 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.921569 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.921694 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.923098 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.924360 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.926344 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.927012 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.927168 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.931248 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.931255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.931259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.931302 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.952315 (+ 21013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.953314 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.953321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.953325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.953490 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.953910 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.953915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.953919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.954288 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:34.954527 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:34.955009 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 16:02:34.955272 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":472,"thread_start_us":301,"threads_started":1}
I20260924 16:02:34.956094 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.935390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.16.233.190:34359)
0924 16:02:34.936107 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.936112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.936133 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:02:34.936197 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.936200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.936262 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.936352 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.937620 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.938611 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.939911 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.940584 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.940686 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.945822 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.945829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.945833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.945877 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.952245 (+ 6368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.953318 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.953324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.953326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.953535 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.953988 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.953994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.953998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.954443 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:34.954751 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:34.955072 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 16:02:34.955490 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":589,"spinlock_wait_cycles":14720,"thread_start_us":376,"threads_started":1}
I20260924 16:02:34.957924 31187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:02:34.972134 31038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } attempt: 4
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:02:34.984555 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.958857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:41715 (local address 127.0.0.1:58420)
0924 16:02:34.959202 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.959224 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.959354 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.960723 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.960730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.961286 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.961547 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.961560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.963504 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.963514 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.965586 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.965606 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.967154 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.967162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.967390 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.969401 (+ 2011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.969422 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.972556 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:34.981000 (+ 8444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.981004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.981007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.981685 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.982233 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.982241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.982247 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.982510 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.983417 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:34.983422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:34.983604 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 16:02:34.983775 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 16:02:34.984098 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260924 16:02:34.985738 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.959762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.16.233.189:41715)
0924 16:02:34.960388 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.960392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.960411 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:02:34.960471 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.960476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.960551 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.960666 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.961720 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.963098 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.965769 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.966917 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.968512 (+ 1595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.972867 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.972876 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.972881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.972930 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.980830 (+ 7900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.981868 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.981875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.981879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.982071 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.982681 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.982689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.982700 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.983192 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:34.983337 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:34.984929 (+ 1592us) server_negotiation.cc:299] Negotiation successful
0924 16:02:34.985209 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1460,"server-negotiator.queue_time_us":506,"thread_start_us":296,"threads_started":1}
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
I20260924 16:02:34.988118 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.959457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.16.233.189:41715)
0924 16:02:34.959688 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.959694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.959713 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:02:34.959780 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.959785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.959858 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.960040 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.963107 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.963823 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.965881 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.967103 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.967503 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.975104 (+ 7601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.975112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.975117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.975162 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.984480 (+ 9318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.985476 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.985485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.985489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.985648 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.986272 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.986278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.986281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.986607 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:34.986943 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:34.987278 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 16:02:34.987543 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":81}
I20260924 16:02:34.990545 31187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:02:34.995047 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:02:35.000735 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.972907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40348)
0924 16:02:34.973198 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.973221 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.973342 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.974356 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.974361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.974865 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.975131 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.975157 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.976545 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.976553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.978193 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.978212 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.979826 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.979834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.980107 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.982755 (+ 2648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.982785 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.988512 (+ 5727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:34.996636 (+ 8124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.996645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.996650 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.997287 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.997855 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.997862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.997867 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.998098 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.998959 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:34.998968 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:34.999620 (+ 652us) client_negotiation.cc:769] Sending connection context
0924 16:02:34.999821 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 16:02:35.000268 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 16:02:35.001960 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.973592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.16.233.188:44921)
0924 16:02:34.973778 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.973784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.973802 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:02:34.973856 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.973861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.973972 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.974068 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.975344 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.976343 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.978387 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.979093 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.979232 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.988792 (+ 9560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.988801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.988806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.988854 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.996417 (+ 7563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.997482 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.997489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.997492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.997677 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.998283 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.998288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.998293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.998775 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:35.001070 (+ 2295us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:35.001225 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 16:02:35.001541 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":66}
Sep 24 16:02:34 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
I20260924 16:02:35.009107 31038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df: Bootstrap starting.
I20260924 16:02:35.014966 31038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:35.017004 31038 log.cc:824] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:35.019814 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.991439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40364)
0924 16:02:34.991654 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.991676 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.991805 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.992819 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.992826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.993378 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.993592 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.993605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.995557 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.995566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.997300 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.997321 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.998546 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.998552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.998839 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.000644 (+ 1805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:35.000673 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:35.005323 (+ 4650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:35.013618 (+ 8295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:35.013628 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:35.013633 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:35.014238 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:35.014706 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:35.014713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:35.014718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:35.014891 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:35.018514 (+ 3623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:35.018522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:35.018782 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 16:02:35.018951 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 16:02:35.019336 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":26}
I20260924 16:02:35.021245 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.991767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.16.233.188:44921)
0924 16:02:34.992452 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.992457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.992479 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:02:34.992532 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.992540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.992634 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.993736 (+ 1102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.993977 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.995399 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.997509 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.998344 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.999198 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.005630 (+ 6432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:35.005638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:35.005643 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:35.005693 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:35.012097 (+ 6404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.014428 (+ 2331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.014432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.014435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.014588 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.015032 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.015038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.015044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.015545 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:35.020269 (+ 4724us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:35.020441 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 16:02:35.020713 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":1443,"threads_started":1}
I20260924 16:02:35.024730 31038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df: No bootstrap required, opened a new log
I20260924 16:02:35.028012 31187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e: Bootstrap starting.
I20260924 16:02:35.032716 31187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:35.034493 31187 log.cc:824] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:35.041033 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.003478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.16.233.190:34359)
0924 16:02:35.009443 (+ 5965us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.009450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.013045 (+ 3595us) server_negotiation.cc:407] Connection header received
0924 16:02:35.013121 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.013128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.013267 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.013375 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.021992 (+ 8617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.023381 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.025585 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.026096 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.026258 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.030297 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:35.030386 (+ 89us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:35.030393 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:35.030446 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:35.036419 (+ 5973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.037197 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.037206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.037210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.037414 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.037933 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.037940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.037944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.038429 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:35.040153 (+ 1724us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:35.040346 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 16:02:35.040630 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5761}
I20260924 16:02:35.045107 31187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e: No bootstrap required, opened a new log
I20260924 16:02:35.051122 31038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } }
I20260924 16:02:35.052831 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.010974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.16.233.188:44921)
0924 16:02:35.015322 (+ 4348us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.015332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.026383 (+ 11051us) server_negotiation.cc:407] Connection header received
0924 16:02:35.030908 (+ 4525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.030914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.031003 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.031074 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.033086 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.034477 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.037252 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.038034 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.038150 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.043090 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:35.043097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:35.043101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:35.043151 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:35.048842 (+ 5691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.050026 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.050033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.050036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.050225 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.050779 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.050784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.050787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.051255 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:35.051505 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:35.052174 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 16:02:35.052428 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4167}
I20260924 16:02:35.053319 31038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:35.053880 31038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e36eacd995449a876bbd97c570c2df, State: Initialized, Role: FOLLOWER
I20260924 16:02:35.056264 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.003304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.16.233.189:41715)
0924 16:02:35.003967 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.003974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.018543 (+ 14569us) server_negotiation.cc:407] Connection header received
0924 16:02:35.018751 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.018759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.018860 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.018996 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.026483 (+ 7487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.027740 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.038110 (+ 10370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.039672 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.039831 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.045894 (+ 6063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:35.045908 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:35.045916 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:35.045983 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:35.051517 (+ 5534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.052884 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.053481 (+ 597us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.053485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.053688 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.054579 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.054585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.054589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.054989 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:35.055309 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:35.055564 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 16:02:35.055828 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506}
I20260924 16:02:35.056116 31038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } }
I20260924 16:02:35.057343 31205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.057902 31205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:35.058982 31038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:35.058697 31187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } }
I20260924 16:02:35.059221 31187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:35.059352 31187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b7ca4449154fc0a41778e48a75376e, State: Initialized, Role: FOLLOWER
I20260924 16:02:35.060120 31187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } }
I20260924 16:02:35.061451 31208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.061957 31208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:35.063509 31187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:02:35.064652 30990 tablet.cc:2369] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:02:35.076205 31222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:02:35.076434 31222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 16:02:35.082916 31229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:02:35.083146 31229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:02:35.086365 31113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } attempt: 5
Sep 24 16:02:35 dist-test-slave-xn4h krb5kdc[30955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
I20260924 16:02:35.110608 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.087073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:44921 (local address 127.0.0.1:40382)
0924 16:02:35.087481 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:35.087504 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:35.087623 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:35.088640 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:35.088646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:35.089227 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:35.089451 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.089466 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.091169 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.091176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.092630 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.092651 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.093954 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.093960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.094245 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.096052 (+ 1807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:35.096097 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:35.100798 (+ 4701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:35.107357 (+ 6559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:35.107365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:35.107371 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:35.107990 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:35.108503 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:35.108509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:35.108512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:35.108679 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:35.109505 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:35.109510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:35.109689 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 16:02:35.109851 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 16:02:35.110178 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260924 16:02:35.111797 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.087843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.16.233.188:44921)
0924 16:02:35.088041 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.088047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.088066 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:02:35.088123 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.088128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.088248 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.088350 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.089640 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.090949 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.092833 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.093571 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.093712 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.101068 (+ 7356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:35.101076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:35.101081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:35.101125 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:35.107074 (+ 5949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.108169 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.108177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.108181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.108357 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.108829 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.108838 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.108841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.109251 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:35.109435 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:35.111134 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0924 16:02:35.111397 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:02:35.076743 30969 master_runner.cc:186] Error getting master registration for 127.16.233.189:41715: OK, instance_id { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" instance_seqno: 1790265754349897 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:02:35.089143 31120 master_runner.cc:186] Error getting master registration for 127.16.233.189:41715: OK, instance_id { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" instance_seqno: 1790265754349897 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:02:35.118196 30969 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:02:35.118494 31120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:02:35.119753 31113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14: Bootstrap starting.
I20260924 16:02:35.124289 31113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:35.128316 31113 log.cc:824] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:35.132786 31113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14: No bootstrap required, opened a new log
W20260924 16:02:35.134763 31141 tablet.cc:2369] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:35.144605 31113 raft_consensus.cc:348] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } }
I20260924 16:02:35.145128 31113 raft_consensus.cc:374] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:35.145269 31113 raft_consensus.cc:729] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65efdbd110a4c51984121619c4dfe14, State: Initialized, Role: FOLLOWER
I20260924 16:02:35.146055 31113 consensus_queue.cc:260] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } }
I20260924 16:02:35.147217 31232 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.147750 31232 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:35.148717 31113 sys_catalog.cc:564] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:35.174610 31045 master_runner.cc:263] No leader master found from master e65efdbd110a4c51984121619c4dfe14
I20260924 16:02:35.174795 31045 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 16:02:35.178918 31243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:02:35.179224 31243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:02:35.191677 31208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:35.192050 31208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } }
I20260924 16:02:35.193866 31208 leader_election.cc:288] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e36eacd995449a876bbd97c570c2df (127.16.233.190:34359), e65efdbd110a4c51984121619c4dfe14 (127.16.233.189:41715)
I20260924 16:02:35.195590 31011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61b7ca4449154fc0a41778e48a75376e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e36eacd995449a876bbd97c570c2df" is_pre_election: true
I20260924 16:02:35.196305 31011 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b7ca4449154fc0a41778e48a75376e in term 0.
I20260924 16:02:35.197880 31139 leader_election.cc:302] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b7ca4449154fc0a41778e48a75376e, 71e36eacd995449a876bbd97c570c2df; no voters:
I20260924 16:02:35.199474 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61b7ca4449154fc0a41778e48a75376e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e65efdbd110a4c51984121619c4dfe14" is_pre_election: true
I20260924 16:02:35.200192 31087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b7ca4449154fc0a41778e48a75376e in term 0.
I20260924 16:02:35.201784 31208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:35.201987 31208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:35.202097 31208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:35.206269 31208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } }
I20260924 16:02:35.207976 31011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61b7ca4449154fc0a41778e48a75376e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e36eacd995449a876bbd97c570c2df"
I20260924 16:02:35.208252 31011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:35.212186 31011 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b7ca4449154fc0a41778e48a75376e in term 1.
I20260924 16:02:35.212958 31139 leader_election.cc:302] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b7ca4449154fc0a41778e48a75376e, 71e36eacd995449a876bbd97c570c2df; no voters:
I20260924 16:02:35.221500 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61b7ca4449154fc0a41778e48a75376e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e65efdbd110a4c51984121619c4dfe14"
I20260924 16:02:35.221815 31087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:35.225812 31087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b7ca4449154fc0a41778e48a75376e in term 1.
I20260924 16:02:35.226986 31208 leader_election.cc:288] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [CANDIDATE]: Term 1 election: Requested vote from peers 71e36eacd995449a876bbd97c570c2df (127.16.233.190:34359), e65efdbd110a4c51984121619c4dfe14 (127.16.233.189:41715)
I20260924 16:02:35.227491 31208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:35.227967 31208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [term 1 LEADER]: Becoming Leader. State: Replica: 61b7ca4449154fc0a41778e48a75376e, State: Running, Role: LEADER
I20260924 16:02:35.228905 31208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } }
I20260924 16:02:35.246110 31245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61b7ca4449154fc0a41778e48a75376e. Latest consensus state: current_term: 1 leader_uuid: "61b7ca4449154fc0a41778e48a75376e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.246860 31245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:35.251968 31247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:35.254693 31247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:35.277194 31087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [term 1 FOLLOWER]: Refusing update from remote peer 61b7ca4449154fc0a41778e48a75376e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:35.278690 31245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:35.292165 31011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [term 1 FOLLOWER]: Refusing update from remote peer 61b7ca4449154fc0a41778e48a75376e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:35.293566 31245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:35.327561 31232 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61b7ca4449154fc0a41778e48a75376e. Latest consensus state: current_term: 1 leader_uuid: "61b7ca4449154fc0a41778e48a75376e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.327926 31232 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:35.331564 31249 mvcc.cc:205] Tried to move back new op lower bound from 7332928533599346688 to 7332928533433282560. Current Snapshot: MvccSnapshot[applied={T|T < 7332928533599346688}]
I20260924 16:02:35.337199 31232 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61b7ca4449154fc0a41778e48a75376e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.337523 31232 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:35.338361 31250 mvcc.cc:205] Tried to move back new op lower bound from 7332928533599346688 to 7332928533433282560. Current Snapshot: MvccSnapshot[applied={T|T < 7332928533599346688}]
I20260924 16:02:35.342491 31205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61b7ca4449154fc0a41778e48a75376e. Latest consensus state: current_term: 1 leader_uuid: "61b7ca4449154fc0a41778e48a75376e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.342881 31205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:35.350415 31208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61b7ca4449154fc0a41778e48a75376e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.350834 31208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:35.358439 31245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61b7ca4449154fc0a41778e48a75376e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.359550 31251 mvcc.cc:205] Tried to move back new op lower bound from 7332928533599346688 to 7332928533433282560. Current Snapshot: MvccSnapshot[applied={T|T < 7332928533599346688}]
I20260924 16:02:35.359834 31245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:35.366848 31247 catalog_manager.cc:1347] Generated new cluster ID: 9dc81a21a44a4840a20a3fd093615a84
I20260924 16:02:35.366991 31247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:35.383131 31205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61b7ca4449154fc0a41778e48a75376e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e36eacd995449a876bbd97c570c2df" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34359 } } peers { permanent_uuid: "e65efdbd110a4c51984121619c4dfe14" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 41715 } } peers { permanent_uuid: "61b7ca4449154fc0a41778e48a75376e" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 44921 } } }
I20260924 16:02:35.383400 31205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:35.405040 31247 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:35.411461 31247 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:35.446473 31247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61b7ca4449154fc0a41778e48a75376e: Generated new TSK 0
I20260924 16:02:35.447963 31247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:02:36.078627 31222 catalog_manager.cc:1259] Loaded cluster ID: 9dc81a21a44a4840a20a3fd093615a84
I20260924 16:02:36.078742 31222 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df: loading cluster ID for follower catalog manager: success
I20260924 16:02:36.081396 31222 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df: acquiring CA information for follower catalog manager: success
I20260924 16:02:36.083464 31222 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71e36eacd995449a876bbd97c570c2df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:02:36.181671 31243 catalog_manager.cc:1259] Loaded cluster ID: 9dc81a21a44a4840a20a3fd093615a84
I20260924 16:02:36.181775 31243 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14: loading cluster ID for follower catalog manager: success
I20260924 16:02:36.184383 31243 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14: acquiring CA information for follower catalog manager: success
I20260924 16:02:36.186270 31243 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e65efdbd110a4c51984121619c4dfe14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:02:36.604842 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 30969
I20260924 16:02:36.606186 31032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:36.718698 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 30969
I20260924 16:02:36.740527 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 31045
I20260924 16:02:36.741732 31107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:36.849853 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 31045
I20260924 16:02:36.871869 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 31120
I20260924 16:02:36.873080 31182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:36.985324 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 31120
2026-09-24T16:02:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4983 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:37 dist-test-slave-xn4h krb5kdc[31269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:37 dist-test-slave-xn4h krb5kdc[31269](info): set up 2 sockets
Sep 24 16:02:37 dist-test-slave-xn4h krb5kdc[31269](info): commencing operation
krb5kdc: starting...
W20260924 16:02:38.100489 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:02:38 dist-test-slave-xn4h krb5kdc[31269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:02:38.350123 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34361
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:34361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:38.696899 31282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:38.697242 31282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:38.697415 31282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:38.705124 31282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:38.705242 31282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:38.705291 31282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:38.705377 31282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:38.705444 31282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:38.715406 31282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:34361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:38.716790 31282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:38.718207 31282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:38.731068 31288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:38.731040 31291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:38.731001 31289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:38.731654 31282 server_base.cc:1034] running on GCE node
I20260924 16:02:38.732604 31282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:38.734179 31282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:38.735425 31282 hybrid_clock.cc:648] HybridClock initialized: now 1790265758735370 us; error 49 us; skew 500 ppm
I20260924 16:02:38.735813 31282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:38 dist-test-slave-xn4h krb5kdc[31269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265758, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:38.739724 31282 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:02:38.741385 31282 webserver.cc:460] Webserver started at http://127.16.233.190:41919/ using document root <none> and password file <none>
I20260924 16:02:38.742131 31282 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:38.742286 31282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:38.742662 31282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:38.745826 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f"
format_stamp: "Formatted at 2026-09-24 16:02:38 on dist-test-slave-xn4h"
server_key: "e8f3d5dd6b588fe8a2583e8b557587dc"
server_key_iv: "b0e70d53982a9d613e2de2fa86805dca"
server_key_version: "encryptionkey@0"
I20260924 16:02:38.746739 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f"
format_stamp: "Formatted at 2026-09-24 16:02:38 on dist-test-slave-xn4h"
server_key: "e8f3d5dd6b588fe8a2583e8b557587dc"
server_key_iv: "b0e70d53982a9d613e2de2fa86805dca"
server_key_version: "encryptionkey@0"
I20260924 16:02:38.752317 31282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:02:38.756206 31298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:38.757694 31282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:38.757906 31282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f"
format_stamp: "Formatted at 2026-09-24 16:02:38 on dist-test-slave-xn4h"
server_key: "e8f3d5dd6b588fe8a2583e8b557587dc"
server_key_iv: "b0e70d53982a9d613e2de2fa86805dca"
server_key_version: "encryptionkey@0"
I20260924 16:02:38.758143 31282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:38.787081 31282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:38.792024 31282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:38.792359 31282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:38.811868 31282 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:34361
I20260924 16:02:38.811899 31349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:34361 every 8 connection(s)
I20260924 16:02:38.813273 31282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:38.814939 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 31282
I20260924 16:02:38.815184 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:02:38.816071 17318 external_mini_cluster.cc:1428] Setting key c2d9fff74172a5c2887214a17f5fadf6
I20260924 16:02:38.818173 31350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:02:38 dist-test-slave-xn4h krb5kdc[31269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:38.832477 31350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f: Bootstrap starting.
I20260924 16:02:38.835976 31350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:38.837145 31350 log.cc:824] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:38.840418 31350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f: No bootstrap required, opened a new log
I20260924 16:02:38.841089 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:38.817942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.16.233.190:34361)
0924 16:02:38.818822 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:38.818833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:38.818878 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:02:38.820560 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:38.820587 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:38.820997 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:38.821366 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:38.823360 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.824915 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.826865 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.827563 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.827707 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:38.832921 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:38.832930 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:38.832947 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:38.832994 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:38.836388 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:38.837245 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:38.837253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:38.837260 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:38.837498 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:38.837961 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:38.837965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:38.837970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:38.838688 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:38.839006 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:38.839517 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 16:02:38.839993 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":659,"thread_start_us":434,"threads_started":1}
I20260924 16:02:38.845214 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:02:38.849150 31350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34361 } }
I20260924 16:02:38.849601 31350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:38.849720 31350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc1877df5fe4ffeb2d1025d93d22d5f, State: Initialized, Role: FOLLOWER
I20260924 16:02:38.850374 31350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34361 } }
I20260924 16:02:38.850636 31350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:38.850782 31350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:38.850976 31350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:38.853237 31350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34361 } }
I20260924 16:02:38.853794 31350 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc1877df5fe4ffeb2d1025d93d22d5f; no voters:
I20260924 16:02:38.854487 31350 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:38.854657 31355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:38.855386 31355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [term 1 LEADER]: Becoming Leader. State: Replica: 1cc1877df5fe4ffeb2d1025d93d22d5f, State: Running, Role: LEADER
I20260924 16:02:38.855922 31355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34361 } }
I20260924 16:02:38.856720 31350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:38.858704 31357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc1877df5fe4ffeb2d1025d93d22d5f. Latest consensus state: current_term: 1 leader_uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34361 } } }
I20260924 16:02:38.859241 31357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:38.859202 31356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1877df5fe4ffeb2d1025d93d22d5f" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34361 } } }
I20260924 16:02:38.860289 31356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:38.862032 31362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:38.867919 31362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:38.878125 31362 catalog_manager.cc:1347] Generated new cluster ID: 7766ac2f0ca04c50a39e0e716893808b
I20260924 16:02:38.878278 31362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:38.890091 31362 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:38.892243 31362 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:38.900151 31362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cc1877df5fe4ffeb2d1025d93d22d5f: Generated new TSK 0
I20260924 16:02:38.901086 31362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:02:38.958235 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 31282
I20260924 16:02:38.959690 31345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:39.060318 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 31282
2026-09-24T16:02:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2074 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:39 dist-test-slave-xn4h krb5kdc[31379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:39 dist-test-slave-xn4h krb5kdc[31379](info): set up 2 sockets
Sep 24 16:02:39 dist-test-slave-xn4h krb5kdc[31379](info): commencing operation
krb5kdc: starting...
W20260924 16:02:40.192472 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:02:40 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:02:40.442276 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:44733
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:44733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:40.790537 31392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:40.790860 31392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:40.791057 31392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:40.798640 31392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:40.798764 31392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:40.798866 31392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:40.798971 31392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:40.799059 31392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:40.809420 31392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:44733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:44733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:40.810877 31392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:40.812229 31392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:40.824769 31398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:40.824798 31401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:40.824728 31399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:40.825438 31392 server_base.cc:1034] running on GCE node
I20260924 16:02:40.826543 31392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:40.828066 31392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:40.829320 31392 hybrid_clock.cc:648] HybridClock initialized: now 1790265760829262 us; error 50 us; skew 500 ppm
I20260924 16:02:40.829704 31392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:40 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:40.833060 31392 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:02:40.834738 31392 webserver.cc:460] Webserver started at http://127.16.233.190:46671/ using document root <none> and password file <none>
I20260924 16:02:40.835484 31392 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:40.835639 31392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:40.835968 31392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:40.839234 31392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "6a85f36ec8a3413986ba6f5ccc4af850"
format_stamp: "Formatted at 2026-09-24 16:02:40 on dist-test-slave-xn4h"
server_key: "0988b437ffec863c413dcbd0a32170d4"
server_key_iv: "d2481ea9ea275bfbacb63adead7c5403"
server_key_version: "encryptionkey@0"
I20260924 16:02:40.840085 31392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "6a85f36ec8a3413986ba6f5ccc4af850"
format_stamp: "Formatted at 2026-09-24 16:02:40 on dist-test-slave-xn4h"
server_key: "0988b437ffec863c413dcbd0a32170d4"
server_key_iv: "d2481ea9ea275bfbacb63adead7c5403"
server_key_version: "encryptionkey@0"
I20260924 16:02:40.845741 31392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:02:40.849786 31408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:40.851387 31392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:40.851622 31392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "6a85f36ec8a3413986ba6f5ccc4af850"
format_stamp: "Formatted at 2026-09-24 16:02:40 on dist-test-slave-xn4h"
server_key: "0988b437ffec863c413dcbd0a32170d4"
server_key_iv: "d2481ea9ea275bfbacb63adead7c5403"
server_key_version: "encryptionkey@0"
I20260924 16:02:40.851859 31392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:40.867542 31392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:40.871996 31392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:40.872318 31392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:40.892277 31392 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:44733
I20260924 16:02:40.892308 31460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:44733 every 8 connection(s)
I20260924 16:02:40.893812 31392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:40.896713 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 31392
I20260924 16:02:40.896948 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:02:40.897821 17318 external_mini_cluster.cc:1428] Setting key 23a29e1dd5c6ac166b17e1fa890b5afe
I20260924 16:02:40.898487 31461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:40.913859 31461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Bootstrap starting.
Sep 24 16:02:40 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:40.917352 31461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:40.918569 31461 log.cc:824] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:40.921690 31461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: No bootstrap required, opened a new log
I20260924 16:02:40.923026 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:40.900064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.16.233.190:44733)
0924 16:02:40.900879 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:40.900891 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:40.900932 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:02:40.903377 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:40.903404 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:40.903850 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:40.904217 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:40.906156 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.907566 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.909445 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.910176 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.910378 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:40.915298 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:40.915309 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:40.915323 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:40.915369 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:40.918550 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:40.919316 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:40.919324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:40.919334 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:40.919508 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:40.919965 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:40.919971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:40.919976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:40.920537 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:40.920867 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:40.921357 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 16:02:40.921879 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":386,"threads_started":1}
I20260924 16:02:40.929850 31461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:40.930300 31461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:40.930454 31461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a85f36ec8a3413986ba6f5ccc4af850, State: Initialized, Role: FOLLOWER
I20260924 16:02:40.931159 31461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:40.931422 31461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:40.931583 31461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:40.931763 31461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:40.934208 31461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:40.934832 31461 leader_election.cc:302] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a85f36ec8a3413986ba6f5ccc4af850; no voters:
I20260924 16:02:40.935510 31461 leader_election.cc:288] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:40.935787 31466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:40.936805 31466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 1 LEADER]: Becoming Leader. State: Replica: 6a85f36ec8a3413986ba6f5ccc4af850, State: Running, Role: LEADER
I20260924 16:02:40.937500 31466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:40.937909 31461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:40.941128 31467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } } }
I20260924 16:02:40.941556 31467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:40.941803 31468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a85f36ec8a3413986ba6f5ccc4af850. Latest consensus state: current_term: 1 leader_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } } }
I20260924 16:02:40.942121 31468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:40.945182 31475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:40.950129 31475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:40.959916 31475 catalog_manager.cc:1347] Generated new cluster ID: c43757da4a0c49f79d4ca0b249666d5d
I20260924 16:02:40.960098 31475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:40.985513 31475 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:40.987078 31475 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:40.996932 31475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Generated new TSK 0
I20260924 16:02:40.997872 31475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 16:02:41.058068 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:44733
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:41.405788 31487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:41.406117 31487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:41.406221 31487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:41.413708 31487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:41.413818 31487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:41.413936 31487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:02:41.424471 31487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:41.425904 31487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:41.427397 31487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:41.441448 31495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:41.441454 31493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:41.441823 31492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:41.442833 31487 server_base.cc:1034] running on GCE node
I20260924 16:02:41.443602 31487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:41.444641 31487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:41.445919 31487 hybrid_clock.cc:648] HybridClock initialized: now 1790265761445861 us; error 42 us; skew 500 ppm
I20260924 16:02:41.446282 31487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:41 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265761, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:41.449788 31487 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:02:41.451470 31487 webserver.cc:460] Webserver started at http://127.16.233.129:38097/ using document root <none> and password file <none>
I20260924 16:02:41.452280 31487 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:41.452416 31487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:41.452759 31487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:41.456087 31487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "c6cfb32b93da4c15a657edd0b891f377"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "75781cf63bcafd5d87c159ae2c136770"
server_key_iv: "b1ff475e18123ba5cba104c38f238b98"
server_key_version: "encryptionkey@0"
I20260924 16:02:41.457095 31487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "c6cfb32b93da4c15a657edd0b891f377"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "75781cf63bcafd5d87c159ae2c136770"
server_key_iv: "b1ff475e18123ba5cba104c38f238b98"
server_key_version: "encryptionkey@0"
I20260924 16:02:41.463433 31487 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 16:02:41.467344 31502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:41.468950 31487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:41.469173 31487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "c6cfb32b93da4c15a657edd0b891f377"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "75781cf63bcafd5d87c159ae2c136770"
server_key_iv: "b1ff475e18123ba5cba104c38f238b98"
server_key_version: "encryptionkey@0"
I20260924 16:02:41.469395 31487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:41.492834 31487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:41.497774 31487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:41.498103 31487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:41.499259 31487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:41.500761 31487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:41.500844 31487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:41.501010 31487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:41.501077 31487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:41.536208 31487 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:42621
I20260924 16:02:41.536263 31616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:42621 every 8 connection(s)
I20260924 16:02:41.537611 31487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:41.543895 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 31487
I20260924 16:02:41.544188 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:41.544945 17318 external_mini_cluster.cc:1428] Setting key 5f5236dc11e0d777adeb738406394d5a
Sep 24 16:02:41 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265761, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:41.559823 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:41.540555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:37635 (local address 127.16.233.190:44733)
0924 16:02:41.541117 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:41.541122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:41.542183 (+ 1061us) server_negotiation.cc:407] Connection header received
0924 16:02:41.544198 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:41.544207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:41.544317 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:41.544454 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:41.546246 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.547055 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.548892 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.549316 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.549456 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:41.553371 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:41.553377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:41.553381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:41.553416 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:41.556326 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:41.557190 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:41.557196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:41.557201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:41.557341 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:41.557858 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:41.557863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:41.557868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:41.558168 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:41.558367 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:41.559165 (+ 798us) server_negotiation.cc:299] Negotiation successful
0924 16:02:41.559427 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":233,"threads_started":1}
I20260924 16:02:41.560981 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:41.541183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:37635)
0924 16:02:41.541978 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:41.542025 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:41.543865 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:41.544676 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:41.544684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:41.545302 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:41.546054 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:41.546081 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.547254 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.547259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:41.548728 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:41.548742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.549588 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.549596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:41.549841 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:41.550625 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:41.550648 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:41.553095 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:41.556519 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:41.556530 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:41.556557 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:41.557047 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:41.557470 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:41.557475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:41.557479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:41.557731 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:41.558339 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:41.558349 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:41.558746 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 16:02:41.559041 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 16:02:41.559671 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":350,"threads_started":1}
I20260924 16:02:41.562940 31617 heartbeater.cc:347] Connected to a master server at 127.16.233.190:44733
I20260924 16:02:41.563340 31617 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:41.564352 31617 heartbeater.cc:511] Master 127.16.233.190:44733 requested a full tablet report, sending...
I20260924 16:02:41.567274 31425 ts_manager.cc:194] Registered new tserver with Master: c6cfb32b93da4c15a657edd0b891f377 (127.16.233.129:42621)
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 16:02:41.569567 31425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:37635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 16:02:41.601656 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:44733
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:41.942503 31623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:41.942869 31623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:41.943042 31623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:41.950681 31623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:41.950847 31623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:41.951035 31623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:02:41.962113 31623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:41.963737 31623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:41.965171 31623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:41.980073 31629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:41.980073 31628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:41.980334 31631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:41.981132 31623 server_base.cc:1034] running on GCE node
I20260924 16:02:41.981796 31623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:41.982944 31623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:41.984210 31623 hybrid_clock.cc:648] HybridClock initialized: now 1790265761984163 us; error 47 us; skew 500 ppm
I20260924 16:02:41.984568 31623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:41 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265761, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:41.988049 31623 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:02:41.989650 31623 webserver.cc:460] Webserver started at http://127.16.233.130:42197/ using document root <none> and password file <none>
I20260924 16:02:41.990494 31623 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:41.990653 31623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:41.990983 31623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:41.994127 31623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "c137ca62be754bd9861753e02389d4ea"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "84f2f7271dda18caaa6821d706dbd491"
server_key_iv: "3128f631f3837c67da27fe3e2689c21a"
server_key_version: "encryptionkey@0"
I20260924 16:02:41.995085 31623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "c137ca62be754bd9861753e02389d4ea"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "84f2f7271dda18caaa6821d706dbd491"
server_key_iv: "3128f631f3837c67da27fe3e2689c21a"
server_key_version: "encryptionkey@0"
I20260924 16:02:42.000914 31623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 16:02:42.004709 31638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:42.006114 31623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:02:42.006348 31623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "c137ca62be754bd9861753e02389d4ea"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "84f2f7271dda18caaa6821d706dbd491"
server_key_iv: "3128f631f3837c67da27fe3e2689c21a"
server_key_version: "encryptionkey@0"
I20260924 16:02:42.006587 31623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:42.028939 31623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:42.033375 31623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:42.033707 31623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:42.034859 31623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:42.036363 31623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:42.036470 31623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:42.036633 31623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:42.036739 31623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:42.072031 31623 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:41473
I20260924 16:02:42.072130 31752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:41473 every 8 connection(s)
I20260924 16:02:42.073493 31623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:42.075515 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 31623
I20260924 16:02:42.075721 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:42.076237 17318 external_mini_cluster.cc:1428] Setting key aed8dd0d37f032e080420bfd2cf1febb
Sep 24 16:02:42 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265761, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:42.099562 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.076754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:39385 (local address 127.16.233.190:44733)
0924 16:02:42.077279 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.077283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.080454 (+ 3171us) server_negotiation.cc:407] Connection header received
0924 16:02:42.080517 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.080520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.080606 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.080698 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:42.083074 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.083822 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.086696 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.087151 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.087299 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.091665 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:42.091671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:42.091674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:42.091718 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:42.094958 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:42.096316 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:42.096321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:42.096324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:42.096489 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:42.097153 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:42.097157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:42.097160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:42.097555 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:42.097744 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:42.098978 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0924 16:02:42.099248 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":282,"threads_started":1}
I20260924 16:02:42.101481 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.077413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:39385)
0924 16:02:42.078283 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.078358 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.080132 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.081046 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.081062 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.081896 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:42.082841 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.082878 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.084015 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.084023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.086506 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.086524 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.087423 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.087430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.087703 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.088450 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:42.088476 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:42.091312 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:42.095199 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:42.095217 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:42.095244 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:42.096112 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:42.096649 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:42.096658 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:42.096665 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:42.097012 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:42.097737 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:42.097750 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:42.098431 (+ 681us) client_negotiation.cc:769] Sending connection context
0924 16:02:42.098868 (+ 437us) client_negotiation.cc:240] Negotiation successful
0924 16:02:42.099675 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":364,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
I20260924 16:02:42.103416 31753 heartbeater.cc:347] Connected to a master server at 127.16.233.190:44733
I20260924 16:02:42.103858 31753 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:42.104753 31753 heartbeater.cc:511] Master 127.16.233.190:44733 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
I20260924 16:02:42.106588 31425 ts_manager.cc:194] Registered new tserver with Master: c137ca62be754bd9861753e02389d4ea (127.16.233.130:41473)
I20260924 16:02:42.107477 31425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:39385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 16:02:42.138739 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:44733
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:42.486438 31759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:42.486815 31759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:42.487001 31759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:42.494644 31759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:42.494828 31759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:42.495034 31759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:02:42.505273 31759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:42.506695 31759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:42.508036 31759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:42.521291 31765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:42.521339 31764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:42.521311 31767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:42.522615 31759 server_base.cc:1034] running on GCE node
I20260924 16:02:42.523343 31759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:42.524338 31759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:42.525558 31759 hybrid_clock.cc:648] HybridClock initialized: now 1790265762525506 us; error 38 us; skew 500 ppm
I20260924 16:02:42.525920 31759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:42 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265762, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:42.529381 31759 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:02:42.531122 31759 webserver.cc:460] Webserver started at http://127.16.233.131:38797/ using document root <none> and password file <none>
I20260924 16:02:42.531845 31759 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:42.531955 31759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:42.532231 31759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:42.535516 31759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "37d45156be4a4e34b4f59e488f4257bc"
format_stamp: "Formatted at 2026-09-24 16:02:42 on dist-test-slave-xn4h"
server_key: "c915f8d17d745469277f9ebb9788d16e"
server_key_iv: "e782f10e639ff9def063a710ab02da0d"
server_key_version: "encryptionkey@0"
I20260924 16:02:42.536619 31759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "37d45156be4a4e34b4f59e488f4257bc"
format_stamp: "Formatted at 2026-09-24 16:02:42 on dist-test-slave-xn4h"
server_key: "c915f8d17d745469277f9ebb9788d16e"
server_key_iv: "e782f10e639ff9def063a710ab02da0d"
server_key_version: "encryptionkey@0"
I20260924 16:02:42.543756 31759 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260924 16:02:42.547600 31774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:42.549132 31759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:42.549353 31759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "37d45156be4a4e34b4f59e488f4257bc"
format_stamp: "Formatted at 2026-09-24 16:02:42 on dist-test-slave-xn4h"
server_key: "c915f8d17d745469277f9ebb9788d16e"
server_key_iv: "e782f10e639ff9def063a710ab02da0d"
server_key_version: "encryptionkey@0"
I20260924 16:02:42.549587 31759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:42.569319 31759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:42.573494 31617 heartbeater.cc:503] Master 127.16.233.190:44733 was elected leader, sending a full tablet report...
I20260924 16:02:42.575590 31759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:42.575937 31759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:42.576954 31759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:42.578461 31759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:42.578539 31759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:42.578707 31759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:42.578759 31759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:42.612064 31759 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:36685
I20260924 16:02:42.612109 31889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:36685 every 8 connection(s)
I20260924 16:02:42.613410 31759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:42.623253 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 31759
I20260924 16:02:42.623464 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:42.624019 17318 external_mini_cluster.cc:1428] Setting key e33fd2fb575e7e430d55b491bda2fb44
Sep 24 16:02:42 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265762, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:42.635095 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.615922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:47103 (local address 127.16.233.190:44733)
0924 16:02:42.616403 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.616408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.617299 (+ 891us) server_negotiation.cc:407] Connection header received
0924 16:02:42.619228 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.619234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.619343 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.619468 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:42.621066 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.621860 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.623812 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.624602 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.624773 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.629111 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:42.629116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:42.629120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:42.629182 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:42.631850 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:42.632719 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:42.632723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:42.632726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:42.632868 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:42.633349 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:42.633354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:42.633358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:42.633671 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:42.633804 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:42.634583 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 16:02:42.634779 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":255,"threads_started":1}
I20260924 16:02:42.636091 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.616395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:47103)
0924 16:02:42.617127 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.617173 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.618935 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.619728 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.619737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.620270 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:42.620868 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.620886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.622029 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.622039 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.623629 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.623643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.624932 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.624941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.625288 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.626139 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:42.626172 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:42.628878 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:42.632026 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:42.632034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:42.632049 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:42.632533 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:42.632970 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:42.632973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:42.632976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:42.633216 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:42.633794 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:42.633802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:42.634167 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 16:02:42.634476 (+ 309us) client_negotiation.cc:240] Negotiation successful
0924 16:02:42.634930 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":275,"threads_started":1}
I20260924 16:02:42.637557 31890 heartbeater.cc:347] Connected to a master server at 127.16.233.190:44733
I20260924 16:02:42.637948 31890 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:42.639110 31890 heartbeater.cc:511] Master 127.16.233.190:44733 requested a full tablet report, sending...
I20260924 16:02:42.640897 31425 ts_manager.cc:194] Registered new tserver with Master: 37d45156be4a4e34b4f59e488f4257bc (127.16.233.131:36685)
I20260924 16:02:42.641832 31425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:47103
I20260924 16:02:42.652889 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:02:42.671118 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.657675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.16.233.190:44733)
0924 16:02:42.657843 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.657848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.658379 (+ 531us) server_negotiation.cc:407] Connection header received
0924 16:02:42.658548 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.658551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.658622 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.658713 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:42.660189 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.660918 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.662205 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.662705 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.662808 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.665266 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:42.665272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:42.665275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:42.665313 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:42.668061 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:42.668869 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:42.668873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:42.668875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:42.669025 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:42.669479 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:42.669483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:42.669486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:42.669838 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:42.670153 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:42.670519 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 16:02:42.670774 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:02:42.675453 31425 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:42.682682 31425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:42.719671 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.696303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.16.233.129:42621)
0924 16:02:42.709637 (+ 13334us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.709651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.709695 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:02:42.709841 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.709854 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.710189 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.710796 (+ 607us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:42.712072 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.713908 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.716740 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.718265 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.718551 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.718760 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:42.719098 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 16:02:42.719296 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13072,"thread_start_us":14114,"threads_started":1}
I20260924 16:02:42.719805 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.696028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:42621 (local address 127.0.0.1:49432)
0924 16:02:42.707154 (+ 11126us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.707182 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.707360 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.711180 (+ 3820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.711192 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.711236 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:42.711799 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.711816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.714081 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.714093 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.716546 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.716569 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.718560 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.718568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.718842 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.718887 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:02:42.719052 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:02:42.719307 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10959,"mutex_wait_us":28,"thread_start_us":10791,"threads_started":1}
I20260924 16:02:42.722995 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.695923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41473 (local address 127.0.0.1:35714)
0924 16:02:42.707073 (+ 11150us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.707120 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.707338 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.715118 (+ 7780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.715123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.715154 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:42.715646 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.715664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.717449 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.717457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.719449 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.719469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.722039 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.722046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.722248 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.722281 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:02:42.722416 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 16:02:42.722557 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10316,"mutex_wait_us":59,"thread_start_us":17870,"threads_started":1}
I20260924 16:02:42.723024 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.696112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.16.233.130:41473)
0924 16:02:42.714592 (+ 18480us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.714600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.714626 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:02:42.714706 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.714712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.714881 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.715877 (+ 996us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:42.716151 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.717309 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.719667 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.721810 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.721989 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.722169 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:42.722486 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 16:02:42.722671 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18286,"thread_start_us":18121,"threads_started":1}
I20260924 16:02:42.723886 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.698854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:36685 (local address 127.0.0.1:42434)
0924 16:02:42.714122 (+ 15268us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.714147 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.714385 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.715844 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.715849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.715875 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:42.716322 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.716333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.718285 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.718291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.720709 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.720725 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.722907 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.722913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.723273 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.723310 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:02:42.723407 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:02:42.723576 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15086,"thread_start_us":7194,"threads_started":1}
I20260924 16:02:42.724052 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.698941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.16.233.131:36685)
0924 16:02:42.713495 (+ 14554us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.713503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.714254 (+ 751us) server_negotiation.cc:407] Connection header received
0924 16:02:42.714630 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.714639 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.714880 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.715444 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:42.716539 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.718129 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.720912 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.722352 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.722579 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.722747 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:42.723473 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 16:02:42.723640 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14375,"thread_start_us":14551,"threads_started":1}
I20260924 16:02:42.724365 31551 tablet_service.cc:1467] Processing CreateTablet for tablet 74f274a0730040449296f56ea3180c62 (DEFAULT_TABLE table=test-table [id=c8906e6e16e54d80ad57b34ee99540a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:42.725855 31551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f274a0730040449296f56ea3180c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:42.727083 31686 tablet_service.cc:1467] Processing CreateTablet for tablet 74f274a0730040449296f56ea3180c62 (DEFAULT_TABLE table=test-table [id=c8906e6e16e54d80ad57b34ee99540a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:42.727795 31823 tablet_service.cc:1467] Processing CreateTablet for tablet 74f274a0730040449296f56ea3180c62 (DEFAULT_TABLE table=test-table [id=c8906e6e16e54d80ad57b34ee99540a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:42.728564 31686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f274a0730040449296f56ea3180c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:42.729223 31823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f274a0730040449296f56ea3180c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:42.738857 31909 tablet_bootstrap.cc:492] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: Bootstrap starting.
I20260924 16:02:42.742719 31909 tablet_bootstrap.cc:654] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:42.743845 31909 log.cc:824] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:42.745355 31910 tablet_bootstrap.cc:492] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: Bootstrap starting.
I20260924 16:02:42.747395 31909 tablet_bootstrap.cc:492] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: No bootstrap required, opened a new log
I20260924 16:02:42.747394 31912 tablet_bootstrap.cc:492] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: Bootstrap starting.
I20260924 16:02:42.747704 31909 ts_tablet_manager.cc:1397] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260924 16:02:42.749323 31910 tablet_bootstrap.cc:654] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:42.750679 31910 log.cc:824] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:42.751075 31912 tablet_bootstrap.cc:654] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:42.752354 31912 log.cc:824] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:42.754186 31910 tablet_bootstrap.cc:492] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: No bootstrap required, opened a new log
I20260924 16:02:42.754539 31910 ts_tablet_manager.cc:1397] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260924 16:02:42.755599 31912 tablet_bootstrap.cc:492] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: No bootstrap required, opened a new log
I20260924 16:02:42.755897 31912 ts_tablet_manager.cc:1397] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:02:42.758786 31909 raft_consensus.cc:348] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } }
I20260924 16:02:42.759300 31909 raft_consensus.cc:374] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:42.759418 31909 raft_consensus.cc:729] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cfb32b93da4c15a657edd0b891f377, State: Initialized, Role: FOLLOWER
I20260924 16:02:42.760139 31909 consensus_queue.cc:260] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } }
I20260924 16:02:42.763744 31909 ts_tablet_manager.cc:1428] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260924 16:02:42.765218 31910 raft_consensus.cc:348] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } }
I20260924 16:02:42.765645 31910 raft_consensus.cc:374] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:42.765427 31912 raft_consensus.cc:348] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } }
I20260924 16:02:42.765780 31910 raft_consensus.cc:729] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d45156be4a4e34b4f59e488f4257bc, State: Initialized, Role: FOLLOWER
I20260924 16:02:42.765899 31912 raft_consensus.cc:374] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:42.766008 31912 raft_consensus.cc:729] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c137ca62be754bd9861753e02389d4ea, State: Initialized, Role: FOLLOWER
I20260924 16:02:42.766628 31912 consensus_queue.cc:260] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } }
I20260924 16:02:42.766616 31910 consensus_queue.cc:260] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } }
I20260924 16:02:42.767969 31890 heartbeater.cc:503] Master 127.16.233.190:44733 was elected leader, sending a full tablet report...
I20260924 16:02:42.768618 31910 ts_tablet_manager.cc:1428] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 16:02:42.769553 31753 heartbeater.cc:503] Master 127.16.233.190:44733 was elected leader, sending a full tablet report...
I20260924 16:02:42.770371 31912 ts_tablet_manager.cc:1428] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
W20260924 16:02:42.790416 31618 tablet.cc:2369] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:02:42.825992 31754 tablet.cc:2369] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:02:42.865038 31891 tablet.cc:2369] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:42.868831 31915 raft_consensus.cc:482] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:42.869273 31915 raft_consensus.cc:504] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } }
I20260924 16:02:42.871323 31915 leader_election.cc:288] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c137ca62be754bd9861753e02389d4ea (127.16.233.130:41473), 37d45156be4a4e34b4f59e488f4257bc (127.16.233.131:36685)
I20260924 16:02:42.878798 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.871695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:58143 (local address 127.16.233.130:41473)
0924 16:02:42.871938 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.871944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.872902 (+ 958us) server_negotiation.cc:407] Connection header received
0924 16:02:42.873170 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.873179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.873258 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.873328 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:42.874087 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.874908 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.876882 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.877681 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.877813 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.877916 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:42.878361 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 16:02:42.878516 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:02:42.878865 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.871922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:40569 (local address 127.16.233.131:36685)
0924 16:02:42.872145 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.872153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.873011 (+ 858us) server_negotiation.cc:407] Connection header received
0924 16:02:42.873221 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.873225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.873286 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.873402 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:42.874123 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.874922 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.876912 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.877776 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.877945 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.878049 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:42.878458 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 16:02:42.878599 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:02:42.879006 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.871905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:36685 (local address 127.16.233.129:40569)
0924 16:02:42.872883 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.872898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.873078 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.873434 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.873439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.873467 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:42.873920 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.873932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.875094 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.875098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.876715 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.876728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.878051 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.878058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.878219 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.878252 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:02:42.878426 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 16:02:42.878571 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":81,"thread_start_us":552,"threads_started":1}
I20260924 16:02:42.879402 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.871730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41473 (local address 127.16.233.129:58143)
0924 16:02:42.872773 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.872791 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.872979 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.873388 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.873393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.873419 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:42.873909 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.873920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.875057 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.875065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.876693 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.876708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.877920 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.877928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.878145 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.878174 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:02:42.878289 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 16:02:42.878453 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":3,"thread_start_us":540,"threads_started":1}
I20260924 16:02:42.880162 31843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f274a0730040449296f56ea3180c62" candidate_uuid: "c6cfb32b93da4c15a657edd0b891f377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d45156be4a4e34b4f59e488f4257bc" is_pre_election: true
I20260924 16:02:42.880328 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f274a0730040449296f56ea3180c62" candidate_uuid: "c6cfb32b93da4c15a657edd0b891f377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c137ca62be754bd9861753e02389d4ea" is_pre_election: true
I20260924 16:02:42.880729 31843 raft_consensus.cc:2454] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cfb32b93da4c15a657edd0b891f377 in term 0.
I20260924 16:02:42.880877 31706 raft_consensus.cc:2454] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cfb32b93da4c15a657edd0b891f377 in term 0.
I20260924 16:02:42.881582 31504 leader_election.cc:302] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37d45156be4a4e34b4f59e488f4257bc, c6cfb32b93da4c15a657edd0b891f377; no voters:
I20260924 16:02:42.882085 31915 raft_consensus.cc:2781] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:42.882249 31915 raft_consensus.cc:482] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:42.882400 31915 raft_consensus.cc:3037] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:42.884636 31915 raft_consensus.cc:504] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } }
I20260924 16:02:42.885298 31915 leader_election.cc:288] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [CANDIDATE]: Term 1 election: Requested vote from peers c137ca62be754bd9861753e02389d4ea (127.16.233.130:41473), 37d45156be4a4e34b4f59e488f4257bc (127.16.233.131:36685)
I20260924 16:02:42.885881 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f274a0730040449296f56ea3180c62" candidate_uuid: "c6cfb32b93da4c15a657edd0b891f377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c137ca62be754bd9861753e02389d4ea"
I20260924 16:02:42.885921 31843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f274a0730040449296f56ea3180c62" candidate_uuid: "c6cfb32b93da4c15a657edd0b891f377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d45156be4a4e34b4f59e488f4257bc"
I20260924 16:02:42.886124 31706 raft_consensus.cc:3037] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:42.886224 31843 raft_consensus.cc:3037] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:42.888562 31706 raft_consensus.cc:2454] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6cfb32b93da4c15a657edd0b891f377 in term 1.
I20260924 16:02:42.888563 31843 raft_consensus.cc:2454] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6cfb32b93da4c15a657edd0b891f377 in term 1.
I20260924 16:02:42.889134 31503 leader_election.cc:302] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c137ca62be754bd9861753e02389d4ea, c6cfb32b93da4c15a657edd0b891f377; no voters:
I20260924 16:02:42.889544 31915 raft_consensus.cc:2781] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:42.890077 31915 raft_consensus.cc:686] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 1 LEADER]: Becoming Leader. State: Replica: c6cfb32b93da4c15a657edd0b891f377, State: Running, Role: LEADER
I20260924 16:02:42.890647 31915 consensus_queue.cc:237] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } }
I20260924 16:02:42.895875 31424 catalog_manager.cc:5495] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6cfb32b93da4c15a657edd0b891f377 (127.16.233.129). New cstate: current_term: 1 leader_uuid: "c6cfb32b93da4c15a657edd0b891f377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:42.959239 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.953542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.16.233.129:42621)
0924 16:02:42.953784 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.953793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.953819 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:02:42.953930 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.953933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.954014 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.954202 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:42.954719 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.955439 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.957032 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.957443 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.957545 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.958031 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:42.958646 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:42.958760 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:42.958836 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 16:02:42.958915 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:02:42.968474 31706 raft_consensus.cc:1261] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [term 1 FOLLOWER]: Refusing update from remote peer c6cfb32b93da4c15a657edd0b891f377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:42.968477 31843 raft_consensus.cc:1261] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [term 1 FOLLOWER]: Refusing update from remote peer c6cfb32b93da4c15a657edd0b891f377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:42.969575 31920 consensus_queue.cc:1035] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:42.970042 31915 consensus_queue.cc:1035] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:42.989315 31924 mvcc.cc:205] Tried to move back new op lower bound from 7332928565107654656 to 7332928564804116480. Current Snapshot: MvccSnapshot[applied={T|T < 7332928565107654656}]
I20260924 16:02:42.996752 31424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:42.997280 31424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:43.001720 31424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Sending DeleteTablet for 3 replicas of tablet 74f274a0730040449296f56ea3180c62
I20260924 16:02:43.003017 31686 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f274a0730040449296f56ea3180c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:42 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:35714
I20260924 16:02:43.003196 31551 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f274a0730040449296f56ea3180c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:42 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:49432
I20260924 16:02:43.004040 31932 tablet_replica.cc:330] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: stopping tablet replica
I20260924 16:02:43.004494 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 31487
I20260924 16:02:43.004433 31823 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f274a0730040449296f56ea3180c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:42 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:42434
I20260924 16:02:43.004555 31932 raft_consensus.cc:2229] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:43.005236 31932 raft_consensus.cc:2258] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:43.008073 31932 ts_tablet_manager.cc:1905] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:43.016357 31934 tablet_replica.cc:330] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: stopping tablet replica
I20260924 16:02:43.016813 31934 raft_consensus.cc:2229] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:43.017354 31934 raft_consensus.cc:2258] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:43.017637 31932 ts_tablet_manager.cc:1918] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:43.017907 31932 log.cc:1196] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/74f274a0730040449296f56ea3180c62
I20260924 16:02:43.018615 31932 ts_tablet_manager.cc:1939] T 74f274a0730040449296f56ea3180c62 P c137ca62be754bd9861753e02389d4ea: Deleting consensus metadata
I20260924 16:02:43.020829 31409 catalog_manager.cc:4841] TS c137ca62be754bd9861753e02389d4ea (127.16.233.130:41473): tablet 74f274a0730040449296f56ea3180c62 (table test-table [id=c8906e6e16e54d80ad57b34ee99540a4]) successfully deleted
I20260924 16:02:43.023126 31933 tablet_replica.cc:330] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: stopping tablet replica
I20260924 16:02:43.023713 31933 raft_consensus.cc:2229] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:43.024498 31933 raft_consensus.cc:2258] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:43.024686 31934 ts_tablet_manager.cc:1905] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:43.028205 31933 ts_tablet_manager.cc:1905] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 16:02:43 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 16:02:43.032560 31934 ts_tablet_manager.cc:1918] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:43.032763 31934 log.cc:1196] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/74f274a0730040449296f56ea3180c62
I20260924 16:02:43.033344 31934 ts_tablet_manager.cc:1939] T 74f274a0730040449296f56ea3180c62 P 37d45156be4a4e34b4f59e488f4257bc: Deleting consensus metadata
I20260924 16:02:43.035482 31410 catalog_manager.cc:4841] TS 37d45156be4a4e34b4f59e488f4257bc (127.16.233.131:36685): tablet 74f274a0730040449296f56ea3180c62 (table test-table [id=c8906e6e16e54d80ad57b34ee99540a4]) successfully deleted
I20260924 16:02:43.037724 31933 ts_tablet_manager.cc:1918] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:43.037967 31933 log.cc:1196] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/74f274a0730040449296f56ea3180c62
I20260924 16:02:43.038645 31933 ts_tablet_manager.cc:1939] T 74f274a0730040449296f56ea3180c62 P c6cfb32b93da4c15a657edd0b891f377: Deleting consensus metadata
I20260924 16:02:43.040822 31412 catalog_manager.cc:4841] TS c6cfb32b93da4c15a657edd0b891f377 (127.16.233.129:42621): tablet 74f274a0730040449296f56ea3180c62 (table test-table [id=c8906e6e16e54d80ad57b34ee99540a4]) successfully deleted
I20260924 16:02:43.041558 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:43.005665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.16.233.129:42621)
0924 16:02:43.005861 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:43.005870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:43.018405 (+ 12535us) server_negotiation.cc:407] Connection header received
0924 16:02:43.018955 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:43.018961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:43.019113 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:43.019211 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:43.021461 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.022646 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.024708 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.025328 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.025417 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:43.032161 (+ 6744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:43.032182 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:43.032201 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:43.032272 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:43.037081 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:43.038378 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:43.038389 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:43.038392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:43.038740 (+ 348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:43.039352 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:43.039359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:43.039362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:43.039863 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:43.040218 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:43.040916 (+ 698us) server_negotiation.cc:299] Negotiation successful
0924 16:02:43.041168 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:02:43.042354 31612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:43.159577 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 31487
I20260924 16:02:43.178823 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 31623
Sep 24 16:02:43 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:02:43.194878 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:43.179670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.16.233.130:41473)
0924 16:02:43.179858 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:43.179866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:43.179889 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:02:43.180118 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:43.180124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:43.180200 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:43.180306 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:43.181590 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.182400 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.183771 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.184164 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.184302 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:43.188811 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:43.188828 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:43.188843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:43.188891 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:43.191812 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:43.192601 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:43.192618 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:43.192620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:43.192757 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:43.193152 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:43.193160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:43.193164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:43.193560 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:43.193907 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:43.194236 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 16:02:43.194496 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:02:43.195580 31748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:43.287175 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 31623
I20260924 16:02:43.305094 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 31759
Sep 24 16:02:43 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 16:02:43.319706 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:43.305894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.16.233.131:36685)
0924 16:02:43.306083 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:43.306087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:43.306107 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:02:43.306262 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:43.306267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:43.306412 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:43.306499 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:43.307748 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.308456 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.309689 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.310057 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.310153 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:43.314497 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:43.314514 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:43.314531 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:43.314572 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:43.317162 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:43.317816 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:43.317823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:43.317825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:43.317960 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:43.318294 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:43.318298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:43.318299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:43.318738 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:43.318979 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:43.319238 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 16:02:43.319417 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:02:43.320400 31885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:43.410688 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 31759
I20260924 16:02:43.427662 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 31392
I20260924 16:02:43.428836 31456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:43.542793 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 31392
I20260924 16:02:43.565279 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:44733
--webserver_interface=127.16.233.190
--webserver_port=46671
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:44733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:43.912920 31940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:43.913259 31940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:43.913350 31940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:43.920799 31940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:43.920928 31940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:43.920975 31940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:43.921042 31940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:43.921095 31940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:43.931347 31940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:44733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:44733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=46671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:43.932735 31940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:43.934068 31940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:43.961365 31949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:43.961337 31947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:43.961423 31946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:43.962275 31940 server_base.cc:1034] running on GCE node
I20260924 16:02:43.963302 31940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:43.965036 31940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:43.966296 31940 hybrid_clock.cc:648] HybridClock initialized: now 1790265763966241 us; error 41 us; skew 500 ppm
I20260924 16:02:43.966830 31940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:43 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265763, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:43.974944 31940 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:02:43.977039 31940 webserver.cc:460] Webserver started at http://127.16.233.190:46671/ using document root <none> and password file <none>
I20260924 16:02:43.977931 31940 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:43.978124 31940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:43.984956 31940 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:02:43.988555 31956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:43.990288 31940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:43.990587 31940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "6a85f36ec8a3413986ba6f5ccc4af850"
format_stamp: "Formatted at 2026-09-24 16:02:40 on dist-test-slave-xn4h"
server_key: "0988b437ffec863c413dcbd0a32170d4"
server_key_iv: "d2481ea9ea275bfbacb63adead7c5403"
server_key_version: "encryptionkey@0"
I20260924 16:02:43.991788 31940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:44.024950 31940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:44.029611 31940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:44.029939 31940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:44.050683 31940 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:44733
I20260924 16:02:44.050705 32007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:44733 every 8 connection(s)
I20260924 16:02:44.052431 31940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:44.060352 32008 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:02:44.061621 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 31940
I20260924 16:02:44.062544 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:42621
--local_ip_for_outbound_sockets=127.16.233.129
--tserver_master_addrs=127.16.233.190:44733
--webserver_port=38097
--webserver_interface=127.16.233.129
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:44.064147 32008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Bootstrap starting.
I20260924 16:02:44.085016 32008 log.cc:824] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:44.098738 32008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:02:44.099327 32008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Bootstrap complete.
I20260924 16:02:44.109179 32008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:44.110136 32008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a85f36ec8a3413986ba6f5ccc4af850, State: Initialized, Role: FOLLOWER
I20260924 16:02:44.110834 32008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:44.111120 32008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:44.111271 32008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:44.111466 32008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:02:44.115824 32008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:44.116520 32008 leader_election.cc:302] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a85f36ec8a3413986ba6f5ccc4af850; no voters:
I20260924 16:02:44.117422 32008 leader_election.cc:288] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:02:44.117841 32012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:02:44.124696 32008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:44.124996 32012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 2 LEADER]: Becoming Leader. State: Replica: 6a85f36ec8a3413986ba6f5ccc4af850, State: Running, Role: LEADER
I20260924 16:02:44.125712 32012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:44.132002 32013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } } }
I20260924 16:02:44.132471 32013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:44.132786 32014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a85f36ec8a3413986ba6f5ccc4af850. Latest consensus state: current_term: 2 leader_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } } }
I20260924 16:02:44.133210 32014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:44.143643 32019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:44.156221 32019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:44.157513 32019 catalog_manager.cc:1259] Loaded cluster ID: c43757da4a0c49f79d4ca0b249666d5d
I20260924 16:02:44.157658 32019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:44.162194 32019 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 16:02:44.189467 32019 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 16:02:44.205822 32019 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:44.207347 32019 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Loaded TSK: 0
I20260924 16:02:44.208570 32019 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:02:44.468865 32010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:44.469228 32010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:44.469383 32010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:44.477237 32010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:44.477419 32010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:44.477600 32010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:02:44.488287 32010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:42621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=38097
--tserver_master_addrs=127.16.233.190:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:44.489809 32010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:44.491281 32010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:44.507517 32036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:44.507548 32037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:44.508401 32039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:44.508915 32010 server_base.cc:1034] running on GCE node
I20260924 16:02:44.509588 32010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:44.510632 32010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:44.511873 32010 hybrid_clock.cc:648] HybridClock initialized: now 1790265764511812 us; error 56 us; skew 500 ppm
I20260924 16:02:44.512197 32010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:44 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265764, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:44.515501 32010 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:02:44.517125 32010 webserver.cc:460] Webserver started at http://127.16.233.129:38097/ using document root <none> and password file <none>
I20260924 16:02:44.517851 32010 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:44.518019 32010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:44.524057 32010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 16:02:44.531628 32046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:44.534168 32010 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.004s
I20260924 16:02:44.534415 32010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "c6cfb32b93da4c15a657edd0b891f377"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "75781cf63bcafd5d87c159ae2c136770"
server_key_iv: "b1ff475e18123ba5cba104c38f238b98"
server_key_version: "encryptionkey@0"
I20260924 16:02:44.535588 32010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:44.574760 32010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:44.580171 32010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:44.580473 32010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:44.581750 32010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:44.583309 32010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:44.583415 32010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:44.583567 32010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:44.583664 32010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:44.618005 32010 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:42621
I20260924 16:02:44.618208 32160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:42621 every 8 connection(s)
I20260924 16:02:44.619629 32010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:44.622264 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 32010
I20260924 16:02:44.622988 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:41473
--local_ip_for_outbound_sockets=127.16.233.130
--tserver_master_addrs=127.16.233.190:44733
--webserver_port=42197
--webserver_interface=127.16.233.130
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:02:44 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265764, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:44.648998 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:44.623200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:39939)
0924 16:02:44.624126 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:44.624185 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:44.626282 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:44.628074 (+ 1792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:44.628081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:44.628612 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:44.629185 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:44.629204 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.631175 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.631183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:44.633459 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:44.633481 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.634946 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.634956 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:44.635280 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:44.636077 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:44.636113 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:44.638988 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:44.644415 (+ 5427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:44.644423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:44.644437 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:44.644941 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:44.645567 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:44.645571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:44.645573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:44.645795 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:44.646916 (+ 1121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:44.646923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:44.647257 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 16:02:44.647499 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 16:02:44.647942 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":398,"threads_started":1}
I20260924 16:02:44.649722 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:44.622827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:39939 (local address 127.16.233.190:44733)
0924 16:02:44.623484 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:44.623505 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:44.624345 (+ 840us) server_negotiation.cc:407] Connection header received
0924 16:02:44.626959 (+ 2614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:44.626982 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:44.627506 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:44.627947 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:44.629460 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.630977 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.633634 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.634287 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.634544 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:44.639369 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:44.639384 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:44.639411 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:44.639478 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:44.644207 (+ 4729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:44.645119 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:44.645132 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:44.645144 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:44.645415 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:44.645920 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:44.645928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:44.645933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:44.646757 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:44.646988 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:44.647827 (+ 839us) server_negotiation.cc:299] Negotiation successful
0924 16:02:44.648350 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":468,"threads_started":1}
I20260924 16:02:44.652781 32161 heartbeater.cc:347] Connected to a master server at 127.16.233.190:44733
I20260924 16:02:44.653131 32161 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:44.654062 32161 heartbeater.cc:511] Master 127.16.233.190:44733 requested a full tablet report, sending...
I20260924 16:02:44.656674 31973 ts_manager.cc:194] Registered new tserver with Master: c6cfb32b93da4c15a657edd0b891f377 (127.16.233.129:42621)
I20260924 16:02:44.658196 31973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:39939
W20260924 16:02:44.975328 32165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:44.975766 32165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:44.975955 32165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:44.983912 32165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:44.984061 32165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:44.984184 32165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:02:44.994567 32165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:41473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=42197
--tserver_master_addrs=127.16.233.190:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:44.995998 32165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:44.997432 32165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:45.011199 32170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:45.011258 32171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:45.011380 32173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:45.012490 32165 server_base.cc:1034] running on GCE node
I20260924 16:02:45.013113 32165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:45.014093 32165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:45.015319 32165 hybrid_clock.cc:648] HybridClock initialized: now 1790265765015277 us; error 50 us; skew 500 ppm
I20260924 16:02:45.015659 32165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:45 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265765, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:45.018874 32165 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:02:45.020447 32165 webserver.cc:460] Webserver started at http://127.16.233.130:42197/ using document root <none> and password file <none>
I20260924 16:02:45.021142 32165 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:45.021266 32165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:45.028280 32165 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:02:45.033401 32180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:45.035758 32165 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 16:02:45.035974 32165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "c137ca62be754bd9861753e02389d4ea"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "84f2f7271dda18caaa6821d706dbd491"
server_key_iv: "3128f631f3837c67da27fe3e2689c21a"
server_key_version: "encryptionkey@0"
I20260924 16:02:45.037887 32165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:45.063966 32165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:45.068591 32165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:45.068907 32165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:45.070282 32165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:45.071867 32165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:45.071959 32165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:45.072047 32165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:45.072106 32165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:45.108310 32165 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:41473
I20260924 16:02:45.108361 32295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:41473 every 8 connection(s)
I20260924 16:02:45.109802 32165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:45.111380 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 32165
I20260924 16:02:45.111970 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:36685
--local_ip_for_outbound_sockets=127.16.233.131
--tserver_master_addrs=127.16.233.190:44733
--webserver_port=38797
--webserver_interface=127.16.233.131
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:02:45 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265765, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:45.138479 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.112354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:58819 (local address 127.16.233.190:44733)
0924 16:02:45.112618 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.112627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.117378 (+ 4751us) server_negotiation.cc:407] Connection header received
0924 16:02:45.119695 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.119700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.119809 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.119905 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:45.122211 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.122986 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.125830 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.126436 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.126584 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.131642 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:45.131647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:45.131651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:45.131701 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:45.134511 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:45.135678 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:45.135682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:45.135685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:45.135827 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:45.136417 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:45.136421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:45.136423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:45.136733 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:45.136847 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:45.137929 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0924 16:02:45.138144 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 16:02:45.140053 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.112914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:58819)
0924 16:02:45.117154 (+ 4240us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:45.117215 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:45.119308 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:45.120217 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:45.120231 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:45.121072 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:45.121964 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.121997 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.123197 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.123206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.125611 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.125634 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.126693 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.126702 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.127074 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.127865 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:45.127902 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:45.131331 (+ 3429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:45.134704 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:45.134720 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:45.134744 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:45.135520 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:45.135964 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:45.135972 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:45.135979 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:45.136287 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:45.136892 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:45.136904 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:45.137457 (+ 553us) client_negotiation.cc:769] Sending connection context
0924 16:02:45.137849 (+ 392us) client_negotiation.cc:240] Negotiation successful
0924 16:02:45.138495 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3908,"thread_start_us":3713,"threads_started":1}
I20260924 16:02:45.141830 32296 heartbeater.cc:347] Connected to a master server at 127.16.233.190:44733
I20260924 16:02:45.142191 32296 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:45.143126 32296 heartbeater.cc:511] Master 127.16.233.190:44733 requested a full tablet report, sending...
I20260924 16:02:45.144991 31973 ts_manager.cc:194] Registered new tserver with Master: c137ca62be754bd9861753e02389d4ea (127.16.233.130:41473)
I20260924 16:02:45.146179 31973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:58819
W20260924 16:02:45.458366 32299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:45.458701 32299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:45.458802 32299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:45.466229 32299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:45.466440 32299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:45.466578 32299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:02:45.476872 32299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:36685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=38797
--tserver_master_addrs=127.16.233.190:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:45.478219 32299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:45.479620 32299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:45.493350 32304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:45.493278 32305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:45.493312 32307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:45.494239 32299 server_base.cc:1034] running on GCE node
I20260924 16:02:45.494966 32299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:45.496120 32299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:45.497344 32299 hybrid_clock.cc:648] HybridClock initialized: now 1790265765497288 us; error 48 us; skew 500 ppm
I20260924 16:02:45.497637 32299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:45 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265765, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:45.500890 32299 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:02:45.503078 32299 webserver.cc:460] Webserver started at http://127.16.233.131:38797/ using document root <none> and password file <none>
I20260924 16:02:45.503813 32299 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:45.503974 32299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:45.509322 32299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:02:45.512028 32314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:45.513303 32299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:02:45.513474 32299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "37d45156be4a4e34b4f59e488f4257bc"
format_stamp: "Formatted at 2026-09-24 16:02:42 on dist-test-slave-xn4h"
server_key: "c915f8d17d745469277f9ebb9788d16e"
server_key_iv: "e782f10e639ff9def063a710ab02da0d"
server_key_version: "encryptionkey@0"
I20260924 16:02:45.514460 32299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:45.542179 32299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:45.547171 32299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:45.547480 32299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:45.548583 32299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:45.549988 32299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:45.550086 32299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:45.550170 32299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:45.550226 32299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:45.585541 32299 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:36685
I20260924 16:02:45.585592 32428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:36685 every 8 connection(s)
I20260924 16:02:45.587157 32299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:45.589752 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 32299
Sep 24 16:02:45 dist-test-slave-xn4h krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265765, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:02:45.608510 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.590782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.16.233.190:44733)
0924 16:02:45.591357 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.591366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.591693 (+ 327us) server_negotiation.cc:407] Connection header received
0924 16:02:45.591880 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.591889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.591986 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.592130 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:45.594272 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.595108 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.597137 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.597837 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.597966 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.601617 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:45.601625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:45.601633 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:45.601694 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:45.605184 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:45.605992 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:45.605997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:45.606001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:45.606161 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:45.606622 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:45.606626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:45.606629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:45.607074 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:45.607352 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:45.607746 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 16:02:45.608014 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
I20260924 16:02:45.612924 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.589991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:42443 (local address 127.16.233.190:44733)
0924 16:02:45.590249 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.590256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.592598 (+ 2342us) server_negotiation.cc:407] Connection header received
0924 16:02:45.595216 (+ 2618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.595224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.595310 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.595394 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:45.597654 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.598785 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.601556 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.602297 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.602494 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.606558 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:45.606564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:45.606571 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:45.606613 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:45.609679 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:45.610546 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:45.610550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:45.610552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:45.610662 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:45.611164 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:45.611169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:45.611172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:45.611544 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:45.611701 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:45.612386 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 16:02:45.612592 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 16:02:45.613907 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.590692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:42443)
0924 16:02:45.592405 (+ 1713us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:45.592465 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:45.594929 (+ 2464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:45.595724 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:45.595739 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:45.596588 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:45.597469 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.597498 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.598969 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.598978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.601346 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.601364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.602626 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.602636 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.602968 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.603626 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:45.603650 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:45.606298 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:45.609831 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:45.609839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:45.609852 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:45.610438 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:45.610776 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:45.610780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:45.610784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:45.611031 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:45.611684 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:45.611693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:45.612023 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 16:02:45.612286 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 16:02:45.612836 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"thread_start_us":1140,"threads_started":1}
I20260924 16:02:45.615195 32429 heartbeater.cc:347] Connected to a master server at 127.16.233.190:44733
I20260924 16:02:45.615475 32429 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:45.616142 32429 heartbeater.cc:511] Master 127.16.233.190:44733 requested a full tablet report, sending...
I20260924 16:02:45.617580 31973 ts_manager.cc:194] Registered new tserver with Master: 37d45156be4a4e34b4f59e488f4257bc (127.16.233.131:36685)
I20260924 16:02:45.618803 31973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:42443
I20260924 16:02:45.623878 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:02:45.631098 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.624943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.16.233.190:44733)
0924 16:02:45.625150 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.625156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.625549 (+ 393us) server_negotiation.cc:407] Connection header received
0924 16:02:45.625741 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.625745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.625931 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.626116 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:45.626591 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.627358 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.628831 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.629279 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.629387 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.629935 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:45.630587 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:45.630701 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:45.630797 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 16:02:45.630872 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:02:45.632150 31973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:45.639822 31973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:45.662247 32161 heartbeater.cc:503] Master 127.16.233.190:44733 was elected leader, sending a full tablet report...
I20260924 16:02:45.679597 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.655246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41473 (local address 127.0.0.1:35730)
0924 16:02:45.670568 (+ 15322us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:45.670586 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:45.670801 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:45.671794 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:45.671799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:45.671825 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:45.672177 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.672190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.674055 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.674059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.676662 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.676680 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.678805 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.678811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.678958 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.678987 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:02:45.679068 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 16:02:45.679174 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15143,"mutex_wait_us":189,"thread_start_us":4042,"threads_started":1}
I20260924 16:02:45.679724 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.655487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.16.233.130:41473)
0924 16:02:45.662114 (+ 6627us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.662129 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.670694 (+ 8565us) server_negotiation.cc:407] Connection header received
0924 16:02:45.671033 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.671042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.671318 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.671500 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:45.672373 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.673846 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.676839 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.678232 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.678429 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.678644 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:45.679152 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 16:02:45.679367 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6370,"thread_start_us":6238,"threads_started":1}
I20260924 16:02:45.680207 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.652523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.16.233.131:36685)
0924 16:02:45.669815 (+ 17292us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.669826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.669866 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:02:45.669986 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.669994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.670355 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.671506 (+ 1151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:45.671794 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.673514 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.676658 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.678228 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.679403 (+ 1175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.679592 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:45.679669 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 16:02:45.679853 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17095,"thread_start_us":17311,"threads_started":1}
I20260924 16:02:45.680629 32443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.652360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:36685 (local address 127.0.0.1:42440)
0924 16:02:45.659747 (+ 7387us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:45.659801 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:45.660021 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:45.670684 (+ 10663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:45.670693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:45.670733 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:45.671247 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.671266 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.673672 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.673678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.676483 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.676501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.678495 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.678503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.678700 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.678784 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 16:02:45.678897 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 16:02:45.679126 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7109,"mutex_wait_us":147,"thread_start_us":17839,"threads_started":1}
I20260924 16:02:45.680878 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.653143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.16.233.129:42621)
0924 16:02:45.669282 (+ 16139us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.669295 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.671751 (+ 2456us) server_negotiation.cc:407] Connection header received
0924 16:02:45.671846 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.671855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.672077 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.672274 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:45.672846 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.674068 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.677945 (+ 3877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.679389 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.679600 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.679803 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:45.680365 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 16:02:45.680530 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15858,"thread_start_us":17173,"threads_started":1}
I20260924 16:02:45.682004 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.653133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:42621 (local address 127.0.0.1:49448)
0924 16:02:45.669958 (+ 16825us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:45.669987 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:45.670205 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:45.672269 (+ 2064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:45.672274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:45.672299 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:45.672655 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.672664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.674231 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.674236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.676687 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.676696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.679989 (+ 3293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.679996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.680170 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.680194 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:02:45.680309 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 16:02:45.680456 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16629,"mutex_wait_us":49,"thread_start_us":16409,"threads_started":1}
I20260924 16:02:45.684363 32230 tablet_service.cc:1467] Processing CreateTablet for tablet a3e9c3359b1f4c95937d8169256284c2 (DEFAULT_TABLE table=test-table [id=8fef578b71f44270b0ae5fba415afc23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:45.684585 32363 tablet_service.cc:1467] Processing CreateTablet for tablet a3e9c3359b1f4c95937d8169256284c2 (DEFAULT_TABLE table=test-table [id=8fef578b71f44270b0ae5fba415afc23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:45.685995 32363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e9c3359b1f4c95937d8169256284c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:45.685593 32094 tablet_service.cc:1467] Processing CreateTablet for tablet a3e9c3359b1f4c95937d8169256284c2 (DEFAULT_TABLE table=test-table [id=8fef578b71f44270b0ae5fba415afc23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:45.686594 32230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e9c3359b1f4c95937d8169256284c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:45.687716 32094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e9c3359b1f4c95937d8169256284c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:45.700301 32445 tablet_bootstrap.cc:492] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: Bootstrap starting.
I20260924 16:02:45.703770 32447 tablet_bootstrap.cc:492] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: Bootstrap starting.
I20260924 16:02:45.704866 32445 tablet_bootstrap.cc:654] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:45.706282 32445 log.cc:824] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:45.708007 32447 tablet_bootstrap.cc:654] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:45.709192 32447 log.cc:824] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:45.709672 32445 tablet_bootstrap.cc:492] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: No bootstrap required, opened a new log
I20260924 16:02:45.709987 32445 ts_tablet_manager.cc:1397] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 16:02:45.710031 32446 tablet_bootstrap.cc:492] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: Bootstrap starting.
I20260924 16:02:45.711735 32447 tablet_bootstrap.cc:492] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: No bootstrap required, opened a new log
I20260924 16:02:45.711993 32447 ts_tablet_manager.cc:1397] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 16:02:45.713477 32446 tablet_bootstrap.cc:654] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:45.714781 32446 log.cc:824] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:45.717795 32446 tablet_bootstrap.cc:492] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: No bootstrap required, opened a new log
I20260924 16:02:45.718087 32446 ts_tablet_manager.cc:1397] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260924 16:02:45.720237 32447 raft_consensus.cc:348] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } }
I20260924 16:02:45.720633 32447 raft_consensus.cc:374] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:45.720746 32447 raft_consensus.cc:729] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cfb32b93da4c15a657edd0b891f377, State: Initialized, Role: FOLLOWER
I20260924 16:02:45.720484 32445 raft_consensus.cc:348] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } }
I20260924 16:02:45.720929 32445 raft_consensus.cc:374] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:45.721029 32445 raft_consensus.cc:729] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c137ca62be754bd9861753e02389d4ea, State: Initialized, Role: FOLLOWER
I20260924 16:02:45.721402 32447 consensus_queue.cc:260] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } }
I20260924 16:02:45.721758 32445 consensus_queue.cc:260] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } }
I20260924 16:02:45.723397 32447 ts_tablet_manager.cc:1428] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 16:02:45.729888 32296 heartbeater.cc:503] Master 127.16.233.190:44733 was elected leader, sending a full tablet report...
I20260924 16:02:45.731160 32445 ts_tablet_manager.cc:1428] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
I20260924 16:02:45.731030 32446 raft_consensus.cc:348] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } }
I20260924 16:02:45.731565 32446 raft_consensus.cc:374] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:45.731668 32446 raft_consensus.cc:729] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d45156be4a4e34b4f59e488f4257bc, State: Initialized, Role: FOLLOWER
I20260924 16:02:45.732300 32446 consensus_queue.cc:260] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } }
I20260924 16:02:45.733500 32429 heartbeater.cc:503] Master 127.16.233.190:44733 was elected leader, sending a full tablet report...
I20260924 16:02:45.734094 32446 ts_tablet_manager.cc:1428] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
W20260924 16:02:45.838907 32430 tablet.cc:2369] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:02:45.861933 32297 tablet.cc:2369] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:02:45.872160 32162 tablet.cc:2369] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:46.041106 32451 raft_consensus.cc:482] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:46.041498 32451 raft_consensus.cc:504] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } }
I20260924 16:02:46.043947 32451 leader_election.cc:288] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37d45156be4a4e34b4f59e488f4257bc (127.16.233.131:36685), c137ca62be754bd9861753e02389d4ea (127.16.233.130:41473)
I20260924 16:02:46.051151 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.044540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:34617 (local address 127.16.233.130:41473)
0924 16:02:46.044749 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:46.044757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:46.045900 (+ 1143us) server_negotiation.cc:407] Connection header received
0924 16:02:46.046060 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:46.046069 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:46.046146 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:46.046251 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:46.046841 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.047504 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.049207 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.050050 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.050237 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.050316 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:46.050708 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 16:02:46.050875 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:02:46.051266 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.044559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:56305 (local address 127.16.233.131:36685)
0924 16:02:46.044757 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:46.044762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:46.045476 (+ 714us) server_negotiation.cc:407] Connection header received
0924 16:02:46.045736 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:46.045740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:46.045803 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:46.045911 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:46.046650 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.047339 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.049211 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.050110 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.050268 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.050372 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:46.050853 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 16:02:46.051031 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:02:46.051594 32454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.044500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:41473 (local address 127.16.233.129:34617)
0924 16:02:46.045785 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:46.045800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:46.045907 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:46.046273 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:46.046277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:46.046333 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:46.046688 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.046697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.047656 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.047660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.049052 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.049065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.050265 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.050271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.050501 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.050538 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:02:46.050643 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 16:02:46.050796 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"mutex_wait_us":27,"spinlock_wait_cycles":22400,"thread_start_us":441,"threads_started":1}
I20260924 16:02:46.052052 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.044501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:36685 (local address 127.16.233.129:56305)
0924 16:02:46.045345 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:46.045361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:46.045551 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:46.045996 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:46.046001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:46.046028 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:46.046473 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.046489 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.047480 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.047488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.049042 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.049055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.050500 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.050506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.050679 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.050707 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:02:46.051454 (+ 747us) client_negotiation.cc:240] Negotiation successful
0924 16:02:46.051643 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":52,"thread_start_us":954,"threads_started":1}
I20260924 16:02:46.052875 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e9c3359b1f4c95937d8169256284c2" candidate_uuid: "c6cfb32b93da4c15a657edd0b891f377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c137ca62be754bd9861753e02389d4ea" is_pre_election: true
I20260924 16:02:46.053071 32383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e9c3359b1f4c95937d8169256284c2" candidate_uuid: "c6cfb32b93da4c15a657edd0b891f377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d45156be4a4e34b4f59e488f4257bc" is_pre_election: true
I20260924 16:02:46.053422 32250 raft_consensus.cc:2454] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cfb32b93da4c15a657edd0b891f377 in term 0.
I20260924 16:02:46.053627 32383 raft_consensus.cc:2454] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cfb32b93da4c15a657edd0b891f377 in term 0.
I20260924 16:02:46.054199 32047 leader_election.cc:302] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c137ca62be754bd9861753e02389d4ea, c6cfb32b93da4c15a657edd0b891f377; no voters:
I20260924 16:02:46.054742 32451 raft_consensus.cc:2781] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:46.054915 32451 raft_consensus.cc:482] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:46.055040 32451 raft_consensus.cc:3037] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:46.057392 32451 raft_consensus.cc:504] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } }
I20260924 16:02:46.058182 32451 leader_election.cc:288] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [CANDIDATE]: Term 1 election: Requested vote from peers 37d45156be4a4e34b4f59e488f4257bc (127.16.233.131:36685), c137ca62be754bd9861753e02389d4ea (127.16.233.130:41473)
I20260924 16:02:46.058833 32383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e9c3359b1f4c95937d8169256284c2" candidate_uuid: "c6cfb32b93da4c15a657edd0b891f377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d45156be4a4e34b4f59e488f4257bc"
I20260924 16:02:46.059077 32383 raft_consensus.cc:3037] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:46.061362 32383 raft_consensus.cc:2454] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6cfb32b93da4c15a657edd0b891f377 in term 1.
I20260924 16:02:46.061825 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e9c3359b1f4c95937d8169256284c2" candidate_uuid: "c6cfb32b93da4c15a657edd0b891f377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c137ca62be754bd9861753e02389d4ea"
I20260924 16:02:46.061928 32048 leader_election.cc:302] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37d45156be4a4e34b4f59e488f4257bc, c6cfb32b93da4c15a657edd0b891f377; no voters:
I20260924 16:02:46.062100 32250 raft_consensus.cc:3037] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:46.062321 32451 raft_consensus.cc:2781] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:46.066565 32250 raft_consensus.cc:2454] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6cfb32b93da4c15a657edd0b891f377 in term 1.
I20260924 16:02:46.066879 32451 raft_consensus.cc:686] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 1 LEADER]: Becoming Leader. State: Replica: c6cfb32b93da4c15a657edd0b891f377, State: Running, Role: LEADER
I20260924 16:02:46.067855 32451 consensus_queue.cc:237] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } }
I20260924 16:02:46.072561 31973 catalog_manager.cc:5495] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6cfb32b93da4c15a657edd0b891f377 (127.16.233.129). New cstate: current_term: 1 leader_uuid: "c6cfb32b93da4c15a657edd0b891f377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6cfb32b93da4c15a657edd0b891f377" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 42621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:46.143019 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.137139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.16.233.129:42621)
0924 16:02:46.137351 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:46.137356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:46.137608 (+ 252us) server_negotiation.cc:407] Connection header received
0924 16:02:46.137797 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:46.137801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:46.137882 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:46.138013 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:46.138497 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.139286 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.140683 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.141154 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.141255 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.141753 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:46.142291 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:46.142489 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:46.142608 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 16:02:46.142682 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:02:46.152478 32250 raft_consensus.cc:1261] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [term 1 FOLLOWER]: Refusing update from remote peer c6cfb32b93da4c15a657edd0b891f377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:46.152478 32383 raft_consensus.cc:1261] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [term 1 FOLLOWER]: Refusing update from remote peer c6cfb32b93da4c15a657edd0b891f377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:46.153822 32456 consensus_queue.cc:1035] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c137ca62be754bd9861753e02389d4ea" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:46.154258 32451 consensus_queue.cc:1035] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d45156be4a4e34b4f59e488f4257bc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:46.182601 31972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:46.182924 31972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:46.186825 31972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Sending DeleteTablet for 3 replicas of tablet a3e9c3359b1f4c95937d8169256284c2
I20260924 16:02:46.188104 32363 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e9c3359b1f4c95937d8169256284c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:46 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:42440
I20260924 16:02:46.188177 32094 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e9c3359b1f4c95937d8169256284c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:46 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:49448
I20260924 16:02:46.188930 32230 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e9c3359b1f4c95937d8169256284c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:46 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:35730
I20260924 16:02:46.189085 32469 tablet_replica.cc:330] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: stopping tablet replica
I20260924 16:02:46.189523 32469 raft_consensus.cc:2229] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:46.189949 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 32010
I20260924 16:02:46.190079 32469 raft_consensus.cc:2258] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:46.192458 32469 ts_tablet_manager.cc:1905] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:46.200585 32471 tablet_replica.cc:330] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: stopping tablet replica
I20260924 16:02:46.201309 32471 raft_consensus.cc:2229] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:46.201958 32471 raft_consensus.cc:2258] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:46.202350 32469 ts_tablet_manager.cc:1918] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:46.202530 32469 log.cc:1196] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/a3e9c3359b1f4c95937d8169256284c2
I20260924 16:02:46.203178 32469 ts_tablet_manager.cc:1939] T a3e9c3359b1f4c95937d8169256284c2 P 37d45156be4a4e34b4f59e488f4257bc: Deleting consensus metadata
I20260924 16:02:46.204766 32470 tablet_replica.cc:330] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: stopping tablet replica
I20260924 16:02:46.205243 32470 raft_consensus.cc:2229] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:46.205287 31958 catalog_manager.cc:4841] TS 37d45156be4a4e34b4f59e488f4257bc (127.16.233.131:36685): tablet a3e9c3359b1f4c95937d8169256284c2 (table test-table [id=8fef578b71f44270b0ae5fba415afc23]) successfully deleted
I20260924 16:02:46.205984 32471 ts_tablet_manager.cc:1905] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:46.205983 32470 raft_consensus.cc:2258] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:46.209981 32470 ts_tablet_manager.cc:1905] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:46.213083 32471 ts_tablet_manager.cc:1918] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:46.213236 32471 log.cc:1196] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/a3e9c3359b1f4c95937d8169256284c2
I20260924 16:02:46.213817 32471 ts_tablet_manager.cc:1939] T a3e9c3359b1f4c95937d8169256284c2 P c137ca62be754bd9861753e02389d4ea: Deleting consensus metadata
I20260924 16:02:46.215802 31957 catalog_manager.cc:4841] TS c137ca62be754bd9861753e02389d4ea (127.16.233.130:41473): tablet a3e9c3359b1f4c95937d8169256284c2 (table test-table [id=8fef578b71f44270b0ae5fba415afc23]) successfully deleted
I20260924 16:02:46.217555 32470 ts_tablet_manager.cc:1918] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:46.217743 32470 log.cc:1196] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/a3e9c3359b1f4c95937d8169256284c2
I20260924 16:02:46.218336 32470 ts_tablet_manager.cc:1939] T a3e9c3359b1f4c95937d8169256284c2 P c6cfb32b93da4c15a657edd0b891f377: Deleting consensus metadata
I20260924 16:02:46.220207 31960 catalog_manager.cc:4841] TS c6cfb32b93da4c15a657edd0b891f377 (127.16.233.129:42621): tablet a3e9c3359b1f4c95937d8169256284c2 (table test-table [id=8fef578b71f44270b0ae5fba415afc23]) successfully deleted
I20260924 16:02:46.224357 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.191303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.16.233.129:42621)
0924 16:02:46.191497 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:46.191503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:46.204255 (+ 12752us) server_negotiation.cc:407] Connection header received
0924 16:02:46.204488 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:46.204493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:46.204614 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:46.204741 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:46.208374 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.209584 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.212389 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.213165 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.213276 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.217216 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:46.217237 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:46.217257 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:46.217311 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:46.221183 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:46.221970 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:46.221978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:46.221981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:46.222129 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:46.222631 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:46.222635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:46.222637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:46.222983 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:46.223323 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:46.223686 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 16:02:46.223950 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:02:46.225132 32156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:46.327128 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 32010
I20260924 16:02:46.346539 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 32165
I20260924 16:02:46.361225 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.347445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.16.233.130:41473)
0924 16:02:46.347643 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:46.347651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:46.347672 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:02:46.347883 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:46.347888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:46.347964 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:46.348105 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:46.349690 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.350541 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.351835 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.352251 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.352391 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.355251 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:46.355267 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:46.355283 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:46.355322 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:46.358055 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:46.358955 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:46.358965 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:46.358967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:46.359109 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:46.359522 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:46.359529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:46.359536 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:46.359934 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:46.360270 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:46.360612 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 16:02:46.360846 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:02:46.361907 32291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:46.454216 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 32165
I20260924 16:02:46.474123 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 32299
I20260924 16:02:46.488721 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.474983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.16.233.131:36685)
0924 16:02:46.475197 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:46.475203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:46.475224 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:02:46.475403 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:46.475409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:46.475474 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:46.475611 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:46.477096 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.477877 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.479392 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.479798 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.479965 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.482725 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:46.482743 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:46.482759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:46.482797 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:46.485572 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:46.486553 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:46.486561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:46.486564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:46.486699 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:46.487116 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:46.487124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:46.487127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:46.487488 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:46.487794 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:46.488141 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 16:02:46.488363 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:02:46.489419 32424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:46.580338 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 32299
I20260924 16:02:46.598726 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 31940
I20260924 16:02:46.599992 32003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:02:46.715608 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 31940
I20260924 16:02:46.738412 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:44733
--webserver_interface=127.16.233.190
--webserver_port=46671
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:44733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:47.088531 32477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:47.088883 32477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:47.089069 32477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:47.096664 32477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:47.096794 32477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:47.096890 32477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:47.097007 32477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:47.097096 32477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:47.107573 32477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:44733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:44733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=46671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:47.109035 32477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:47.110438 32477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:47.122540 32483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:47.122550 32485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:47.122608 32482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:47.123564 32477 server_base.cc:1034] running on GCE node
I20260924 16:02:47.124435 32477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:47.126003 32477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:47.127259 32477 hybrid_clock.cc:648] HybridClock initialized: now 1790265767127176 us; error 72 us; skew 500 ppm
I20260924 16:02:47.127656 32477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:47 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:47.131040 32477 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:02:47.132695 32477 webserver.cc:460] Webserver started at http://127.16.233.190:46671/ using document root <none> and password file <none>
I20260924 16:02:47.133450 32477 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:47.133622 32477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:47.140143 32477 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:02:47.145743 32492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:47.147372 32477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:47.147588 32477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "6a85f36ec8a3413986ba6f5ccc4af850"
format_stamp: "Formatted at 2026-09-24 16:02:40 on dist-test-slave-xn4h"
server_key: "0988b437ffec863c413dcbd0a32170d4"
server_key_iv: "d2481ea9ea275bfbacb63adead7c5403"
server_key_version: "encryptionkey@0"
I20260924 16:02:47.148674 32477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:47.168618 32477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:47.173494 32477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:47.173810 32477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:47.193883 32477 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:44733
I20260924 16:02:47.193890 32544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:44733 every 8 connection(s)
I20260924 16:02:47.195637 32477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:47.203125 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 32477
I20260924 16:02:47.203230 32545 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:02:47.204221 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:42621
--local_ip_for_outbound_sockets=127.16.233.129
--tserver_master_addrs=127.16.233.190:44733
--webserver_port=38097
--webserver_interface=127.16.233.129
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:47.208777 32545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Bootstrap starting.
I20260924 16:02:47.231002 32545 log.cc:824] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:47.248579 32545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:02:47.249212 32545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850: Bootstrap complete.
I20260924 16:02:47.259351 32545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:47.260385 32545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a85f36ec8a3413986ba6f5ccc4af850, State: Initialized, Role: FOLLOWER
I20260924 16:02:47.261121 32545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:47.261379 32545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:47.261525 32545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:47.261723 32545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:02:47.265877 32545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:47.266551 32545 leader_election.cc:302] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a85f36ec8a3413986ba6f5ccc4af850; no voters:
I20260924 16:02:47.267292 32545 leader_election.cc:288] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:02:47.267448 32549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:02:47.268640 32549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [term 3 LEADER]: Becoming Leader. State: Replica: 6a85f36ec8a3413986ba6f5ccc4af850, State: Running, Role: LEADER
I20260924 16:02:47.269248 32549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } }
I20260924 16:02:47.269675 32545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:47.272162 32551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a85f36ec8a3413986ba6f5ccc4af850. Latest consensus state: current_term: 3 leader_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } } }
I20260924 16:02:47.272646 32551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:47.273240 32550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f36ec8a3413986ba6f5ccc4af850" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 44733 } } }
I20260924 16:02:47.273578 32550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a85f36ec8a3413986ba6f5ccc4af850 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:47.283313 32554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:47.292572 32554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:47.294384 32554 catalog_manager.cc:1259] Loaded cluster ID: c43757da4a0c49f79d4ca0b249666d5d
I20260924 16:02:47.294515 32554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 16:02:47.297129 32554 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
I20260924 16:02:47.299785 32567 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f0b157979ad google::LogMessage::Fail() at ??:0
@ 0x7f0b1579a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0b157974de google::LogMessage::Flush() at ??:0
@ 0x7f0b15798a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0b2b1ca512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0b2b1c7976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0b2b24f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0b2b24f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0b183e3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0b183e691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0b183e6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0b183c1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0b1b69b6db start_thread at ??:0
@ 0x7f0b1463161f clone at ??:0
W20260924 16:02:47.622191 32547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:47.622591 32547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:47.622758 32547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:47.630441 32547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:47.630565 32547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:47.630787 32547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:02:47.641093 32547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:42621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=38097
--tserver_master_addrs=127.16.233.190:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:47.642545 32547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:47.643934 32547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:47.657723 32588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:47.657791 32590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:47.657764 32587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:47.664137 32547 server_base.cc:1034] running on GCE node
I20260924 16:02:47.664906 32547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:47.665900 32547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:47.667114 32547 hybrid_clock.cc:648] HybridClock initialized: now 1790265767667059 us; error 44 us; skew 500 ppm
I20260924 16:02:47.667479 32547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:47 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:47.670862 32547 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:02:47.672493 32547 webserver.cc:460] Webserver started at http://127.16.233.129:38097/ using document root <none> and password file <none>
I20260924 16:02:47.673269 32547 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:47.673424 32547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:47.679181 32547 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:02:47.682176 32597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:47.683625 32547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:47.683812 32547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "c6cfb32b93da4c15a657edd0b891f377"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "75781cf63bcafd5d87c159ae2c136770"
server_key_iv: "b1ff475e18123ba5cba104c38f238b98"
server_key_version: "encryptionkey@0"
I20260924 16:02:47.684855 32547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:47.711333 32547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:47.716066 32547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:47.716373 32547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:47.717442 32547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:47.718941 32547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:47.719080 32547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:47.719209 32547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:47.719298 32547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:47.754081 32547 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:42621
I20260924 16:02:47.754169 32712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:42621 every 8 connection(s)
I20260924 16:02:47.755859 32547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:47.760761 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:47.758919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:45159)
0924 16:02:47.759651 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:47.759896 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":278,"threads_started":1}
W20260924 16:02:47.761420 32713 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:44733 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.190:44733: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
I20260924 16:02:47.762295 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:47.761926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:55041)
0924 16:02:47.762099 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:47.762191 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 16:02:47.762387 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 32547
I20260924 16:02:47.763096 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:41473
--local_ip_for_outbound_sockets=127.16.233.130
--tserver_master_addrs=127.16.233.190:44733
--webserver_port=42197
--webserver_interface=127.16.233.130
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:47.763327 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:47.762919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:44205)
0924 16:02:47.763060 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:47.763151 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 16:02:47.763571 32713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:02:48.099128 32716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:48.099504 32716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:48.099673 32716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:48.107803 32716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:48.107937 32716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:48.108183 32716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:02:48.121359 32716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:41473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=42197
--tserver_master_addrs=127.16.233.190:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:48.123240 32716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:48.124912 32716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:48.138823 32721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:48.138844 32722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:48.138819 32724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:48.140300 32716 server_base.cc:1034] running on GCE node
I20260924 16:02:48.141006 32716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:48.142031 32716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:48.143251 32716 hybrid_clock.cc:648] HybridClock initialized: now 1790265768143159 us; error 75 us; skew 500 ppm
I20260924 16:02:48.143568 32716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:48 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:48.147099 32716 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:02:48.148769 32716 webserver.cc:460] Webserver started at http://127.16.233.130:42197/ using document root <none> and password file <none>
I20260924 16:02:48.149533 32716 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:48.149670 32716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:48.155848 32716 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:02:48.158922 32731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:48.160497 32716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:02:48.160688 32716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "c137ca62be754bd9861753e02389d4ea"
format_stamp: "Formatted at 2026-09-24 16:02:41 on dist-test-slave-xn4h"
server_key: "84f2f7271dda18caaa6821d706dbd491"
server_key_iv: "3128f631f3837c67da27fe3e2689c21a"
server_key_version: "encryptionkey@0"
I20260924 16:02:48.161762 32716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:48.179442 32716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:48.184535 32716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:48.184808 32716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:48.185804 32716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:48.187340 32716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:48.187428 32716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:48.187507 32716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:48.187580 32716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:48.221577 32716 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:41473
I20260924 16:02:48.221714 376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:41473 every 8 connection(s)
I20260924 16:02:48.223197 32716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:48.227056 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 32716
I20260924 16:02:48.227706 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:36685
--local_ip_for_outbound_sockets=127.16.233.131
--tserver_master_addrs=127.16.233.190:44733
--webserver_port=38797
--webserver_interface=127.16.233.131
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:48.228749 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.226572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:49981)
0924 16:02:48.227495 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.227786 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":389,"threads_started":1}
W20260924 16:02:48.229476 377 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:44733 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.190:44733: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
I20260924 16:02:48.230543 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.230111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:59967)
0924 16:02:48.230280 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.230420 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:02:48.231587 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.231172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:46205)
0924 16:02:48.231345 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.231460 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 16:02:48.232033 377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:02:48.568357 380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:48.568650 380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:48.568827 380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:48.576220 380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:48.576339 380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:48.576525 380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:02:48.586503 380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:36685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=38797
--tserver_master_addrs=127.16.233.190:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:02:48.587877 380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:48.589318 380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:48.603191 386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:48.603536 387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:48.604115 389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:48.605329 380 server_base.cc:1034] running on GCE node
I20260924 16:02:48.605976 380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:48.607061 380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:48.608299 380 hybrid_clock.cc:648] HybridClock initialized: now 1790265768608242 us; error 48 us; skew 500 ppm
I20260924 16:02:48.608635 380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:02:48 dist-test-slave-xn4h krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:48.611896 380 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:02:48.613373 380 webserver.cc:460] Webserver started at http://127.16.233.131:38797/ using document root <none> and password file <none>
I20260924 16:02:48.614125 380 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:48.614277 380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:48.619725 380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 16:02:48.622539 396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:48.623908 380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 16:02:48.624112 380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "37d45156be4a4e34b4f59e488f4257bc"
format_stamp: "Formatted at 2026-09-24 16:02:42 on dist-test-slave-xn4h"
server_key: "c915f8d17d745469277f9ebb9788d16e"
server_key_iv: "e782f10e639ff9def063a710ab02da0d"
server_key_version: "encryptionkey@0"
I20260924 16:02:48.625156 380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:48.645145 380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:48.649147 380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:48.649434 380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:48.650441 380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:48.651810 380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:48.651930 380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:48.652077 380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:48.652166 380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:48.685634 380 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:36685
I20260924 16:02:48.685719 514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:36685 every 8 connection(s)
I20260924 16:02:48.687162 380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:48.692009 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.690125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:51461)
0924 16:02:48.690969 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.691179 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":404,"threads_started":1}
W20260924 16:02:48.692618 515 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:44733 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.190:44733: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
I20260924 16:02:48.693136 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 380
I20260924 16:02:48.693667 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.693168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:47707)
0924 16:02:48.693438 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.693535 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260924 16:02:48.694573 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.694210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:54951)
0924 16:02:48.694373 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.694452 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 16:02:48.694875 515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:02:48.764909 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.764105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:41479)
0924 16:02:48.764567 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.764683 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":225,"threads_started":1}
I20260924 16:02:49.233517 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:49.232655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:54803)
0924 16:02:49.233144 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:49.233283 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":227,"threads_started":1}
I20260924 16:02:49.696249 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:49.695396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:51143)
0924 16:02:49.695897 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:49.696021 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":258,"threads_started":1}
I20260924 16:02:49.766584 522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:49.765763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:42405)
0924 16:02:49.766259 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:49.766417 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":206,"threads_started":1}
I20260924 16:02:50.235406 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.234620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:58427)
0924 16:02:50.235132 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:50.235234 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":238,"threads_started":1}
I20260924 16:02:50.698065 524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.697225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:37989)
0924 16:02:50.697799 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:50.697888 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":254,"threads_started":1}
I20260924 16:02:50.768265 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.767540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:49119)
0924 16:02:50.767985 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:50.768076 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":226,"threads_started":1}
I20260924 16:02:51.237001 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.236295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:40383)
0924 16:02:51.236748 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:51.236828 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
I20260924 16:02:51.699848 527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.699097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:50437)
0924 16:02:51.699580 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:51.699669 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":242,"threads_started":1}
I20260924 16:02:51.770201 528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.769277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:37459)
0924 16:02:51.769876 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:51.770018 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":342,"threads_started":1}
I20260924 16:02:52.238914 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:52.238108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:48589)
0924 16:02:52.238631 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:52.238717 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":289,"threads_started":1}
I20260924 16:02:52.701638 530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:52.700872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:35359)
0924 16:02:52.701332 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:52.701417 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":220,"threads_started":1}
I20260924 16:02:52.771885 531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:52.771165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:58031)
0924 16:02:52.771641 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:52.771721 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":248,"threads_started":1}
I20260924 16:02:53.240737 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.239859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:45065)
0924 16:02:53.240407 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.240531 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":274,"threads_started":1}
I20260924 16:02:53.703236 533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.702578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:45737)
0924 16:02:53.703000 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.703076 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260924 16:02:53.773492 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.772815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:52939)
0924 16:02:53.773248 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.773334 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":228,"threads_started":1}
I20260924 16:02:54.242549 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:54.241702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:55213)
0924 16:02:54.242220 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:54.242379 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":251,"threads_started":1}
I20260924 16:02:54.704883 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:54.704180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:40883)
0924 16:02:54.704628 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:54.704721 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":217,"threads_started":1}
I20260924 16:02:54.775260 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:54.774493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:51775)
0924 16:02:54.774973 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:54.775055 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":199,"threads_started":1}
I20260924 16:02:55.244267 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:55.243477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:57767)
0924 16:02:55.243971 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:55.244054 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":211,"threads_started":1}
I20260924 16:02:55.706741 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:55.705906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:42475)
0924 16:02:55.706429 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:55.706513 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":208,"threads_started":1}
I20260924 16:02:55.776842 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:55.776169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:38707)
0924 16:02:55.776602 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:55.776683 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260924 16:02:56.246212 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:56.245306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:48789)
0924 16:02:56.245820 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:56.245984 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":255,"threads_started":1}
I20260924 16:02:56.708742 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:56.707819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:42661)
0924 16:02:56.708414 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:56.708509 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":338,"threads_started":1}
I20260924 16:02:56.778617 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:56.777788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:54567)
0924 16:02:56.778292 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:56.778434 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":252,"threads_started":1}
I20260924 16:02:57.248152 544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:57.247305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:49071)
0924 16:02:57.247886 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:57.247968 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":318,"threads_started":1}
I20260924 16:02:57.710511 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:57.709661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:41091)
0924 16:02:57.710164 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:57.710247 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260924 16:02:57.780390 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:57.779622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:39889)
0924 16:02:57.780129 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:57.780220 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260924 16:02:58.249804 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:58.249098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:50277)
0924 16:02:58.249554 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:58.249639 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260924 16:02:58.712393 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:58.711569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:53755)
0924 16:02:58.712126 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:58.712215 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":322,"threads_started":1}
I20260924 16:02:58.782018 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:58.781301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:34869)
0924 16:02:58.781734 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:58.781836 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":208,"threads_started":1}
I20260924 16:02:59.251776 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:59.250973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:53929)
0924 16:02:59.251475 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:59.251579 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":250,"threads_started":1}
I20260924 16:02:59.714466 551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:59.713511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:60559)
0924 16:02:59.714102 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:59.714211 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":299,"threads_started":1}
I20260924 16:02:59.783689 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:59.782998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:45485)
0924 16:02:59.783436 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:59.783525 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":221,"threads_started":1}
I20260924 16:03:00.253652 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:00.252911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:40069)
0924 16:03:00.253357 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:00.253489 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":225,"threads_started":1}
I20260924 16:03:00.716109 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:00.715401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:43715)
0924 16:03:00.715843 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:00.715930 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260924 16:03:00.785498 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:00.784664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:39681)
0924 16:03:00.785207 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:00.785306 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":274,"threads_started":1}
I20260924 16:03:01.255554 556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:01.254606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:35733)
0924 16:03:01.255248 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:01.255347 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":373,"threads_started":1}
I20260924 16:03:01.718144 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:01.717253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:44595)
0924 16:03:01.717803 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:01.717904 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":280,"threads_started":1}
I20260924 16:03:01.787240 558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:01.786488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:37995)
0924 16:03:01.786957 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:01.787056 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":227,"threads_started":1}
I20260924 16:03:02.257288 559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:02.256510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:43891)
0924 16:03:02.257029 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:02.257110 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260924 16:03:02.719908 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:02.719179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:49429)
0924 16:03:02.719619 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:02.719706 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":218,"threads_started":1}
I20260924 16:03:02.789238 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:02.788449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.129:42693)
0924 16:03:02.788900 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:02.789059 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":241,"threads_started":1}
I20260924 16:03:03.259500 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:03.258558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.130:33851)
0924 16:03:03.259143 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:03.259299 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":315,"threads_started":1}
I20260924 16:03:03.703732 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 32547
I20260924 16:03:03.721886 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:03.721072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:44733 (local address 127.16.233.131:47793)
0924 16:03:03.721608 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:03.721705 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":278,"threads_started":1}
I20260924 16:03:03.730212 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:03.704800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.16.233.129:42621)
0924 16:03:03.705522 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:03.705540 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:03.705585 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:03:03.708082 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:03.708092 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:03.708517 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:03.708871 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:03.710385 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:03.711619 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:03.712883 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:03.713608 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:03.713882 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:03.723861 (+ 9979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:03.723870 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:03.723887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:03.723926 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:03.726799 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:03.727523 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:03.727530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:03.727540 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:03.727706 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:03.728111 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:03.728117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:03.728120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:03.728736 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:03.729046 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:03.729496 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 16:03:03.729934 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":398,"threads_started":1}
I20260924 16:03:03.731695 32707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:03.815624 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 32547
I20260924 16:03:03.833513 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 32716
I20260924 16:03:03.850488 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:03.834495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.16.233.130:41473)
0924 16:03:03.835037 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:03.835050 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:03.835084 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:03:03.836603 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:03.836612 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:03.837026 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:03.837303 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:03.838728 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:03.839950 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:03.841206 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:03.841848 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:03.841980 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:03.844535 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:03.844542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:03.844556 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:03.844592 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:03.847374 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:03.848022 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:03.848028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:03.848035 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:03.848215 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:03.848539 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:03.848542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:03.848544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:03.849111 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:03.849343 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:03.849701 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 16:03:03.850139 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":254,"threads_started":1}
I20260924 16:03:03.851862 371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:03.935359 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 32716
I20260924 16:03:03.953828 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 380
I20260924 16:03:03.971532 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:03.955004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.16.233.131:36685)
0924 16:03:03.955575 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:03.955588 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:03.955629 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:03:03.957227 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:03.957236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:03.957684 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:03.957972 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:03.959432 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:03.960681 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:03.961919 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:03.962627 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:03.962756 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:03.965534 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:03.965542 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:03.965556 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:03.965603 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:03.968486 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:03.969169 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:03.969176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:03.969181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:03.969331 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:03.969687 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:03.969690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:03.969692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:03.970166 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:03.970473 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:03.970807 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 16:03:03.971228 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":253,"threads_started":1}
I20260924 16:03:03.972880 510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:04.055568 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 380
2026-09-24T16:03:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24996 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:03:04 dist-test-slave-xn4h krb5kdc[575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:03:04 dist-test-slave-xn4h krb5kdc[575](info): set up 2 sockets
Sep 24 16:03:04 dist-test-slave-xn4h krb5kdc[575](info): commencing operation
krb5kdc: starting...
W20260924 16:03:05.189998 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:03:05 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:03:05.441970 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36011
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:03:05.793632 588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:05.793996 588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:05.794167 588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:05.801966 588 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:03:05.802091 588 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:03:05.802160 588 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:03:05.802239 588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:03:05.802367 588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:05.802490 588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:05.802541 588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:03:05.802675 588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:03:05.813503 588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:05.815086 588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:05.816536 588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:05.829002 593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:05.829101 596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:05.829118 594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:05.830379 588 server_base.cc:1034] running on GCE node
I20260924 16:03:05.831265 588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:05.832852 588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:05.834081 588 hybrid_clock.cc:648] HybridClock initialized: now 1790265785833997 us; error 56 us; skew 500 ppm
I20260924 16:03:05.834503 588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:05 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265785, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:05.837880 588 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:03:05.839569 588 webserver.cc:460] Webserver started at http://127.16.233.190:41179/ using document root <none> and password file <none>
I20260924 16:03:05.840349 588 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:05.840446 588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:05.840806 588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:03:05.843986 588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c"
format_stamp: "Formatted at 2026-09-24 16:03:05 on dist-test-slave-xn4h"
server_key: "de38c3302acb03cf6551baeb9762bc26"
server_key_iv: "6fdb84f56464dd5bc748f7570762030e"
server_key_version: "encryptionkey@0"
I20260924 16:03:05.844843 588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c"
format_stamp: "Formatted at 2026-09-24 16:03:05 on dist-test-slave-xn4h"
server_key: "de38c3302acb03cf6551baeb9762bc26"
server_key_iv: "6fdb84f56464dd5bc748f7570762030e"
server_key_version: "encryptionkey@0"
I20260924 16:03:05.850564 588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260924 16:03:05.854440 603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:05.856001 588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:03:05.856240 588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c"
format_stamp: "Formatted at 2026-09-24 16:03:05 on dist-test-slave-xn4h"
server_key: "de38c3302acb03cf6551baeb9762bc26"
server_key_iv: "6fdb84f56464dd5bc748f7570762030e"
server_key_version: "encryptionkey@0"
I20260924 16:03:05.856503 588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:05.878443 588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:05.882907 588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:05.883216 588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:05.903551 588 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:36011
I20260924 16:03:05.903563 656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:36011 every 8 connection(s)
I20260924 16:03:05.905050 588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:03:05.907078 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 588
I20260924 16:03:05.907320 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:03:05.908031 17318 external_mini_cluster.cc:1428] Setting key f412e91a00e129e54f7b90c1bd48960c
I20260924 16:03:05.909457 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:03:05 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:03:05.924659 657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Bootstrap starting.
I20260924 16:03:05.929185 657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:05.930598 657 log.cc:824] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:05.934214 657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: No bootstrap required, opened a new log
I20260924 16:03:05.934588 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:05.910066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.16.233.190:36011)
0924 16:03:05.910919 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:05.910937 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:05.910990 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:03:05.913027 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:05.913056 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:05.913571 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:05.914022 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:05.915448 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:05.917519 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:05.919328 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:05.920516 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:05.920734 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:05.925369 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:05.925380 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:05.925403 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:05.925460 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:05.929769 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:05.930644 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:05.930653 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:05.930663 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:05.930882 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:05.931378 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:05.931384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:05.931387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:05.932030 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:05.932347 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:05.932727 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 16:03:05.933191 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":352,"threads_started":1}
I20260924 16:03:05.943019 657 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:05.943504 657 raft_consensus.cc:374] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:05.943616 657 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0f1d5c8d9e4a2c95d91c1e4ea79b7c, State: Initialized, Role: FOLLOWER
I20260924 16:03:05.944249 657 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:05.944470 657 raft_consensus.cc:388] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:03:05.944643 657 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:03:05.944841 657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:05.947427 657 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:05.948005 657 leader_election.cc:302] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be0f1d5c8d9e4a2c95d91c1e4ea79b7c; no voters:
I20260924 16:03:05.948614 657 leader_election.cc:288] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:03:05.948786 662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:03:05.949515 662 raft_consensus.cc:686] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 1 LEADER]: Becoming Leader. State: Replica: be0f1d5c8d9e4a2c95d91c1e4ea79b7c, State: Running, Role: LEADER
I20260924 16:03:05.950139 662 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:05.950959 657 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:03:05.952986 663 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } } }
I20260924 16:03:05.953498 663 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: This master's current role is: LEADER
I20260924 16:03:05.953816 664 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0f1d5c8d9e4a2c95d91c1e4ea79b7c. Latest consensus state: current_term: 1 leader_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } } }
I20260924 16:03:05.954205 664 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: This master's current role is: LEADER
I20260924 16:03:05.959280 672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:03:05.965332 672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:03:05.975638 672 catalog_manager.cc:1347] Generated new cluster ID: f02556094ae84fada10704aac5581816
I20260924 16:03:05.975787 672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:03:05.994746 672 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:03:05.996723 672 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:03:06.006364 672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Generated new TSK 0
I20260924 16:03:06.007493 672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 16:03:06.079206 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36011
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:03:06.432214 684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:06.432523 684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:06.432703 684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:06.440634 684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:06.440768 684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:06.440972 684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:03:06.451354 684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:06.452806 684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:06.454236 684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:06.469345 689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:06.469429 690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:06.469317 692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:06.475600 684 server_base.cc:1034] running on GCE node
I20260924 16:03:06.476356 684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:06.477451 684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:06.478713 684 hybrid_clock.cc:648] HybridClock initialized: now 1790265786478660 us; error 45 us; skew 500 ppm
I20260924 16:03:06.479116 684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:06 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265786, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:06.482545 684 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:03:06.484242 684 webserver.cc:460] Webserver started at http://127.16.233.129:36287/ using document root <none> and password file <none>
I20260924 16:03:06.484953 684 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:06.485069 684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:06.485345 684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:03:06.488502 684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "e72cbc613172471caed8e4279e1e7008"
format_stamp: "Formatted at 2026-09-24 16:03:06 on dist-test-slave-xn4h"
server_key: "9f6c68d691104e670f557805b7218c04"
server_key_iv: "e0de546aeab8c0f7dbe312e8e73c483a"
server_key_version: "encryptionkey@0"
I20260924 16:03:06.489270 684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "e72cbc613172471caed8e4279e1e7008"
format_stamp: "Formatted at 2026-09-24 16:03:06 on dist-test-slave-xn4h"
server_key: "9f6c68d691104e670f557805b7218c04"
server_key_iv: "e0de546aeab8c0f7dbe312e8e73c483a"
server_key_version: "encryptionkey@0"
I20260924 16:03:06.494962 684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 16:03:06.498967 700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:06.500463 684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:03:06.500662 684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "e72cbc613172471caed8e4279e1e7008"
format_stamp: "Formatted at 2026-09-24 16:03:06 on dist-test-slave-xn4h"
server_key: "9f6c68d691104e670f557805b7218c04"
server_key_iv: "e0de546aeab8c0f7dbe312e8e73c483a"
server_key_version: "encryptionkey@0"
I20260924 16:03:06.500902 684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:06.520414 684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:06.524854 684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:06.525187 684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:06.526273 684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:06.527804 684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:06.527880 684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:06.528079 684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:06.528143 684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:06.563126 684 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:46021
I20260924 16:03:06.563180 814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:46021 every 8 connection(s)
I20260924 16:03:06.564603 684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:03:06.570593 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 684
I20260924 16:03:06.570837 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:03:06.571693 17318 external_mini_cluster.cc:1428] Setting key b54642fcbb3a644d257f522f9d0ba62e
Sep 24 16:03:06 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265786, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:03:06.587339 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:06.567544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:51671 (local address 127.16.233.190:36011)
0924 16:03:06.568041 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:06.568047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:06.568969 (+ 922us) server_negotiation.cc:407] Connection header received
0924 16:03:06.571468 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:06.571473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:06.571555 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:06.571646 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:06.573582 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:06.574424 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:06.576121 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:06.576542 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:06.576698 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:06.580819 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:06.580824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:06.580828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:06.580865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:06.583619 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:06.584686 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:06.584690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:06.584693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:06.584822 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:06.585372 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:06.585375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:06.585377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:06.585669 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:06.585785 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:06.586867 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0924 16:03:06.587076 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":271,"threads_started":1}
I20260924 16:03:06.588972 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:06.568046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:51671)
0924 16:03:06.568787 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:06.568828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:06.571077 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:06.571989 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:06.572003 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:06.572714 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:06.573360 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:06.573380 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:06.574609 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:06.574613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:06.575938 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:06.575951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:06.576804 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:06.576812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:06.577085 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:06.577844 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:06.577875 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:06.580504 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:06.583799 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:06.583812 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:06.583834 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:06.584546 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:06.584952 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:06.584960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:06.584967 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:06.585257 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:06.585807 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:06.585820 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:06.586392 (+ 572us) client_negotiation.cc:769] Sending connection context
0924 16:03:06.586751 (+ 359us) client_negotiation.cc:240] Negotiation successful
0924 16:03:06.587398 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":283,"threads_started":1}
I20260924 16:03:06.590804 815 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36011
I20260924 16:03:06.591178 815 heartbeater.cc:464] Registering TS with master...
I20260924 16:03:06.592154 815 heartbeater.cc:511] Master 127.16.233.190:36011 requested a full tablet report, sending...
I20260924 16:03:06.594820 620 ts_manager.cc:194] Registered new tserver with Master: e72cbc613172471caed8e4279e1e7008 (127.16.233.129:46021)
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
I20260924 16:03:06.596504 620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:51671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 16:03:06.629201 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36011
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:03:06.973462 821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:06.973770 821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:06.973937 821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:06.981355 821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:06.981539 821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:06.981748 821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:03:06.991819 821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:06.993248 821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:06.994820 821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:07.008778 827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:07.008749 829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:07.008816 826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:07.010603 821 server_base.cc:1034] running on GCE node
I20260924 16:03:07.011345 821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:07.012439 821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:07.013680 821 hybrid_clock.cc:648] HybridClock initialized: now 1790265787013628 us; error 49 us; skew 500 ppm
I20260924 16:03:07.013975 821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:07 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265787, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:07.017282 821 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:03:07.018800 821 webserver.cc:460] Webserver started at http://127.16.233.130:41853/ using document root <none> and password file <none>
I20260924 16:03:07.019815 821 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:07.019971 821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:07.020304 821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:03:07.023533 821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "aa9c6aec080f4158b114e5d1ea58e928"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "b3206ee1344e2ec269b0157f0a81b1a1"
server_key_iv: "be5469ebb88ccfe342063322b87ef5c3"
server_key_version: "encryptionkey@0"
I20260924 16:03:07.024334 821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "aa9c6aec080f4158b114e5d1ea58e928"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "b3206ee1344e2ec269b0157f0a81b1a1"
server_key_iv: "be5469ebb88ccfe342063322b87ef5c3"
server_key_version: "encryptionkey@0"
I20260924 16:03:07.029920 821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 16:03:07.033995 836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:07.035540 821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:03:07.035751 821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "aa9c6aec080f4158b114e5d1ea58e928"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "b3206ee1344e2ec269b0157f0a81b1a1"
server_key_iv: "be5469ebb88ccfe342063322b87ef5c3"
server_key_version: "encryptionkey@0"
I20260924 16:03:07.035995 821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:07.071926 821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:07.076229 821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:07.076565 821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:07.077574 821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:07.078969 821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:07.079082 821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:07.079243 821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:07.079320 821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:07.113838 821 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:42079
I20260924 16:03:07.113978 950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:42079 every 8 connection(s)
I20260924 16:03:07.115324 821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:03:07.124485 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 821
I20260924 16:03:07.124696 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:03:07.125165 17318 external_mini_cluster.cc:1428] Setting key 990a44cb1e6404e8439a3f5520ab9b8b
Sep 24 16:03:07 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265787, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:03:07.138600 954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.118109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:33125 (local address 127.16.233.190:36011)
0924 16:03:07.118623 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:07.118630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:07.119660 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 16:03:07.121575 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:07.121580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:07.121652 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:07.121751 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:07.123408 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.124141 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.126108 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.126789 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.126925 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.130927 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:07.130936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:07.130944 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:07.130997 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:07.134852 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:07.135804 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:07.135808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:07.135811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:07.135980 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:07.136518 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:07.136522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:07.136524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:07.136814 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:07.137032 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:07.137927 (+ 895us) server_negotiation.cc:299] Negotiation successful
0924 16:03:07.138185 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":249,"threads_started":1}
I20260924 16:03:07.140110 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.118759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:33125)
0924 16:03:07.119461 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:07.119504 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:07.121305 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:07.121963 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:07.121972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:07.122594 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:07.123233 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.123253 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.124317 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.124328 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.125935 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.125951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.127038 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.127046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.127333 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.128062 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:07.128086 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:07.130646 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:07.135055 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:07.135065 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:07.135081 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:07.135609 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:07.136120 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:07.136125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:07.136129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:07.136363 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:07.136968 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:07.136976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:07.137489 (+ 513us) client_negotiation.cc:769] Sending connection context
0924 16:03:07.137811 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 16:03:07.138487 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
I20260924 16:03:07.141852 952 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36011
I20260924 16:03:07.142220 952 heartbeater.cc:464] Registering TS with master...
I20260924 16:03:07.143028 952 heartbeater.cc:511] Master 127.16.233.190:36011 requested a full tablet report, sending...
I20260924 16:03:07.144678 620 ts_manager.cc:194] Registered new tserver with Master: aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079)
I20260924 16:03:07.145537 620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:33125
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 16:03:07.185274 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36011
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:03:07.521888 958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:07.522243 958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:07.522428 958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:07.529951 958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:07.530140 958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:07.530408 958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:03:07.540992 958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:07.542444 958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:07.543819 958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:07.557359 963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:07.557287 966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:07.557324 964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:07.558151 958 server_base.cc:1034] running on GCE node
I20260924 16:03:07.558833 958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:07.559973 958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:07.561231 958 hybrid_clock.cc:648] HybridClock initialized: now 1790265787561157 us; error 70 us; skew 500 ppm
I20260924 16:03:07.561594 958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:07 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265787, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:07.564978 958 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:03:07.566702 958 webserver.cc:460] Webserver started at http://127.16.233.131:41369/ using document root <none> and password file <none>
I20260924 16:03:07.567451 958 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:07.567556 958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:07.567904 958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:03:07.570967 958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "9bf1c88666ba4eed9b38debf51e5e3ca"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "fab4d3a6c12ea3091e02611f34a9f8c3"
server_key_iv: "eae949b742f96fc96beb3c04195e8912"
server_key_version: "encryptionkey@0"
I20260924 16:03:07.571709 958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "9bf1c88666ba4eed9b38debf51e5e3ca"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "fab4d3a6c12ea3091e02611f34a9f8c3"
server_key_iv: "eae949b742f96fc96beb3c04195e8912"
server_key_version: "encryptionkey@0"
I20260924 16:03:07.576936 958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 16:03:07.580432 973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:07.581733 958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:03:07.581866 958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "9bf1c88666ba4eed9b38debf51e5e3ca"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "fab4d3a6c12ea3091e02611f34a9f8c3"
server_key_iv: "eae949b742f96fc96beb3c04195e8912"
server_key_version: "encryptionkey@0"
I20260924 16:03:07.582041 958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:07.595392 958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:07.599318 958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:07.599553 958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:07.600458 815 heartbeater.cc:503] Master 127.16.233.190:36011 was elected leader, sending a full tablet report...
I20260924 16:03:07.600524 958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:07.602174 958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:07.602319 958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:07.602458 958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:07.602523 958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:07.636071 958 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:34975
I20260924 16:03:07.636113 1088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:34975 every 8 connection(s)
I20260924 16:03:07.637416 958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:03:07.639660 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 958
I20260924 16:03:07.639894 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:03:07.640588 17318 external_mini_cluster.cc:1428] Setting key d09ef98ceb04892334284b351e83d2e9
Sep 24 16:03:07 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265787, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:03:07.657743 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.640197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:37161 (local address 127.16.233.190:36011)
0924 16:03:07.642687 (+ 2490us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:07.642694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:07.642714 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:03:07.643998 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:07.644003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:07.644114 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:07.644232 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:07.645659 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.646503 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.648026 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.648452 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.648594 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.652198 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:07.652203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:07.652206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:07.652239 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:07.654729 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:07.655432 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:07.655439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:07.655442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:07.655572 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:07.656002 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:07.656008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:07.656011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:07.656295 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:07.656463 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:07.657141 (+ 678us) server_negotiation.cc:299] Negotiation successful
0924 16:03:07.657355 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2366,"thread_start_us":2231,"threads_started":1}
I20260924 16:03:07.658771 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.640746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:37161)
0924 16:03:07.641845 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:07.641897 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:07.643740 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:07.644471 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:07.644479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:07.644959 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:07.645492 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.645508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.646663 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.646670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.647868 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.647880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.648684 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.648691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.648920 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.649667 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:07.649688 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:07.652000 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:07.654868 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:07.654877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:07.654894 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:07.655305 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:07.655685 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:07.655690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:07.655694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:07.655880 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:07.656428 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:07.656436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:07.656782 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 16:03:07.657041 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 16:03:07.657549 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":579,"threads_started":1}
I20260924 16:03:07.660092 1089 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36011
I20260924 16:03:07.660436 1089 heartbeater.cc:464] Registering TS with master...
I20260924 16:03:07.661181 1089 heartbeater.cc:511] Master 127.16.233.190:36011 requested a full tablet report, sending...
I20260924 16:03:07.662839 620 ts_manager.cc:194] Registered new tserver with Master: 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975)
I20260924 16:03:07.663928 620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:37161
I20260924 16:03:07.668411 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:03:07.684763 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.672784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.16.233.190:36011)
0924 16:03:07.672981 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:07.672991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:07.673431 (+ 440us) server_negotiation.cc:407] Connection header received
0924 16:03:07.673554 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:07.673558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:07.673632 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:07.673708 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:07.675214 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.675899 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.677302 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.677675 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.677761 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.680066 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:07.680071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:07.680074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:07.680101 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:07.682460 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:07.683076 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:07.683079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:07.683082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:07.683197 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:07.683512 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:07.683515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:07.683517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:07.683821 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:07.684006 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:07.684262 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 16:03:07.684417 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:03:07.688670 620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:03:07.695492 620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:03:07.720609 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.707987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.16.233.129:46021)
0924 16:03:07.713021 (+ 5034us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:07.713031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:07.713060 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:03:07.713181 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:07.713188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:07.713457 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:07.714337 (+ 880us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:07.714589 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.715879 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.717949 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.719441 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.719646 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.719841 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:07.720129 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 16:03:07.720293 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4852,"thread_start_us":4696,"threads_started":1}
I20260924 16:03:07.720621 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.707318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:50620)
0924 16:03:07.712703 (+ 5385us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:07.712731 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:07.712850 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:07.713693 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:07.713700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:07.713728 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:07.714122 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.714135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.716014 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.716020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.717770 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.717785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.719659 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.719667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.719852 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.719909 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 16:03:07.720055 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 16:03:07.720238 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5247,"thread_start_us":5771,"threads_started":1}
I20260924 16:03:07.721892 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.707211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:34975 (local address 127.0.0.1:44082)
0924 16:03:07.708657 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:07.708711 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:07.708881 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:07.714456 (+ 5575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:07.714461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:07.714486 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:07.715048 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.715062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.717200 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.717205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.719279 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.719291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.721065 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.721072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.721238 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.721287 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 16:03:07.721358 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 16:03:07.721482 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"mutex_wait_us":47,"thread_start_us":4001,"threads_started":1}
I20260924 16:03:07.722463 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.708348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:42079 (local address 127.0.0.1:33416)
0924 16:03:07.713330 (+ 4982us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:07.713351 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:07.713546 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:07.714477 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:07.714482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:07.714507 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:07.715105 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.715119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.717050 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.717055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.719631 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:07.719650 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.721519 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.721526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:07.721709 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.721747 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:03:07.721851 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:03:07.722089 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4867,"mutex_wait_us":110,"thread_start_us":4068,"threads_started":1}
I20260924 16:03:07.722607 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.708822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.16.233.130:42079)
0924 16:03:07.712240 (+ 3418us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:07.712253 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:07.713444 (+ 1191us) server_negotiation.cc:407] Connection header received
0924 16:03:07.713758 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:07.713769 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:07.713977 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:07.714151 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:07.715293 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.716876 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.719816 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.721270 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.721484 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.721661 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:07.722077 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 16:03:07.722245 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3256,"thread_start_us":3171,"threads_started":1}
I20260924 16:03:07.723249 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:07.707286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.16.233.131:34975)
0924 16:03:07.713820 (+ 6534us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:07.713827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:07.713860 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:03:07.713964 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:07.713973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:07.714213 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:07.715261 (+ 1048us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:07.715534 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.717052 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.719434 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:07.720856 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:07.722292 (+ 1436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:07.722470 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:07.722539 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 16:03:07.722695 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6383,"thread_start_us":6227,"threads_started":1}
I20260924 16:03:07.725152 748 tablet_service.cc:1467] Processing CreateTablet for tablet 25e6aaf0fc4f4fddb6211b15331a1ff0 (DEFAULT_TABLE table=test-table [id=dac69b5a86044b668f70f9bc2fdfc8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:03:07.726754 748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e6aaf0fc4f4fddb6211b15331a1ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:07.726884 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 25e6aaf0fc4f4fddb6211b15331a1ff0 (DEFAULT_TABLE table=test-table [id=dac69b5a86044b668f70f9bc2fdfc8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:03:07.728284 1022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e6aaf0fc4f4fddb6211b15331a1ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:07.729339 885 tablet_service.cc:1467] Processing CreateTablet for tablet 25e6aaf0fc4f4fddb6211b15331a1ff0 (DEFAULT_TABLE table=test-table [id=dac69b5a86044b668f70f9bc2fdfc8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:03:07.730609 885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e6aaf0fc4f4fddb6211b15331a1ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:07.741334 1108 tablet_bootstrap.cc:492] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: Bootstrap starting.
I20260924 16:03:07.743279 1109 tablet_bootstrap.cc:492] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: Bootstrap starting.
I20260924 16:03:07.745200 1108 tablet_bootstrap.cc:654] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:07.746899 1109 tablet_bootstrap.cc:654] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:07.747077 1110 tablet_bootstrap.cc:492] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: Bootstrap starting.
I20260924 16:03:07.747084 1108 log.cc:824] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:07.748282 1109 log.cc:824] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:07.750409 1110 tablet_bootstrap.cc:654] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:07.750790 1108 tablet_bootstrap.cc:492] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: No bootstrap required, opened a new log
I20260924 16:03:07.751098 1108 ts_tablet_manager.cc:1397] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 16:03:07.751366 1109 tablet_bootstrap.cc:492] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: No bootstrap required, opened a new log
I20260924 16:03:07.751616 1109 ts_tablet_manager.cc:1397] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:03:07.751745 1110 log.cc:824] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:07.754712 1110 tablet_bootstrap.cc:492] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: No bootstrap required, opened a new log
I20260924 16:03:07.754951 1110 ts_tablet_manager.cc:1397] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 16:03:07.759599 1108 raft_consensus.cc:348] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:07.760059 1108 raft_consensus.cc:374] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:07.760180 1108 raft_consensus.cc:729] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa9c6aec080f4158b114e5d1ea58e928, State: Initialized, Role: FOLLOWER
I20260924 16:03:07.760975 1108 consensus_queue.cc:260] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:07.762151 952 heartbeater.cc:503] Master 127.16.233.190:36011 was elected leader, sending a full tablet report...
I20260924 16:03:07.762248 1109 raft_consensus.cc:348] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:07.762713 1109 raft_consensus.cc:374] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:07.762812 1109 raft_consensus.cc:729] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72cbc613172471caed8e4279e1e7008, State: Initialized, Role: FOLLOWER
I20260924 16:03:07.762905 1108 ts_tablet_manager.cc:1428] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260924 16:03:07.763401 1109 consensus_queue.cc:260] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:07.765154 1109 ts_tablet_manager.cc:1428] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260924 16:03:07.767181 1110 raft_consensus.cc:348] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:07.767666 1110 raft_consensus.cc:374] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:07.767810 1110 raft_consensus.cc:729] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf1c88666ba4eed9b38debf51e5e3ca, State: Initialized, Role: FOLLOWER
I20260924 16:03:07.768424 1110 consensus_queue.cc:260] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:07.769975 1089 heartbeater.cc:503] Master 127.16.233.190:36011 was elected leader, sending a full tablet report...
I20260924 16:03:07.770191 1110 ts_tablet_manager.cc:1428] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260924 16:03:07.817286 816 tablet.cc:2369] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:03:07.867491 953 tablet.cc:2369] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:03:07.889107 1090 tablet.cc:2369] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:03:08.001881 1114 raft_consensus.cc:482] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:03:08.002238 1114 raft_consensus.cc:504] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:08.003911 1114 leader_election.cc:288] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975), e72cbc613172471caed8e4279e1e7008 (127.16.233.129:46021)
I20260924 16:03:08.011889 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:08.004588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:58635 (local address 127.16.233.131:34975)
0924 16:03:08.004761 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:08.004767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:08.005545 (+ 778us) server_negotiation.cc:407] Connection header received
0924 16:03:08.005816 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:08.005821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:08.005897 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:08.005994 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:08.006895 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.007612 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.010066 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.010899 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.011100 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:08.011178 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:08.011496 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 16:03:08.011625 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:03:08.011924 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:08.004522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:40713 (local address 127.16.233.129:46021)
0924 16:03:08.004750 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:08.004757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:08.005544 (+ 787us) server_negotiation.cc:407] Connection header received
0924 16:03:08.005770 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:08.005777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:08.005859 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:08.005994 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:08.006822 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.007541 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.010054 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.010933 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.011127 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:08.011206 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:08.011538 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 16:03:08.011667 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:03:08.012099 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:08.004379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:34975 (local address 127.16.233.130:58635)
0924 16:03:08.005417 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:08.005436 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:08.005590 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:08.006049 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:08.006053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:08.006085 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:08.006712 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:08.006724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.007773 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.007778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:08.009898 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:08.009911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.011136 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.011143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:08.011327 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:08.011357 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:03:08.011469 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:03:08.011611 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":157,"spinlock_wait_cycles":24704,"thread_start_us":637,"threads_started":1}
I20260924 16:03:08.012140 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:08.004246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.16.233.130:40713)
0924 16:03:08.005376 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:08.005399 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:08.005590 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:08.006020 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:08.006026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:08.006069 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:08.006658 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:08.006672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.007688 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.007695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:08.009884 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:08.009897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.011170 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.011176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:08.011371 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:08.011394 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:03:08.011505 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 16:03:08.011626 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":150,"spinlock_wait_cycles":36224,"thread_start_us":745,"threads_started":1}
I20260924 16:03:08.013248 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e6aaf0fc4f4fddb6211b15331a1ff0" candidate_uuid: "aa9c6aec080f4158b114e5d1ea58e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" is_pre_election: true
I20260924 16:03:08.013271 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e6aaf0fc4f4fddb6211b15331a1ff0" candidate_uuid: "aa9c6aec080f4158b114e5d1ea58e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72cbc613172471caed8e4279e1e7008" is_pre_election: true
I20260924 16:03:08.013761 769 raft_consensus.cc:2454] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa9c6aec080f4158b114e5d1ea58e928 in term 0.
I20260924 16:03:08.013760 1042 raft_consensus.cc:2454] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa9c6aec080f4158b114e5d1ea58e928 in term 0.
I20260924 16:03:08.014495 837 leader_election.cc:302] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9bf1c88666ba4eed9b38debf51e5e3ca, aa9c6aec080f4158b114e5d1ea58e928, e72cbc613172471caed8e4279e1e7008; no voters:
I20260924 16:03:08.014921 1114 raft_consensus.cc:2781] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:03:08.015084 1114 raft_consensus.cc:482] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:03:08.015193 1114 raft_consensus.cc:3037] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:08.017339 1114 raft_consensus.cc:504] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:08.017946 1114 leader_election.cc:288] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [CANDIDATE]: Term 1 election: Requested vote from peers 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975), e72cbc613172471caed8e4279e1e7008 (127.16.233.129:46021)
I20260924 16:03:08.018451 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e6aaf0fc4f4fddb6211b15331a1ff0" candidate_uuid: "aa9c6aec080f4158b114e5d1ea58e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca"
I20260924 16:03:08.018458 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e6aaf0fc4f4fddb6211b15331a1ff0" candidate_uuid: "aa9c6aec080f4158b114e5d1ea58e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72cbc613172471caed8e4279e1e7008"
I20260924 16:03:08.018671 769 raft_consensus.cc:3037] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:08.018671 1042 raft_consensus.cc:3037] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:08.021934 1042 raft_consensus.cc:2454] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa9c6aec080f4158b114e5d1ea58e928 in term 1.
I20260924 16:03:08.021934 769 raft_consensus.cc:2454] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa9c6aec080f4158b114e5d1ea58e928 in term 1.
I20260924 16:03:08.022440 838 leader_election.cc:302] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf1c88666ba4eed9b38debf51e5e3ca, aa9c6aec080f4158b114e5d1ea58e928; no voters:
I20260924 16:03:08.022853 1114 raft_consensus.cc:2781] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:03:08.023310 1114 raft_consensus.cc:686] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 LEADER]: Becoming Leader. State: Replica: aa9c6aec080f4158b114e5d1ea58e928, State: Running, Role: LEADER
I20260924 16:03:08.023795 1114 consensus_queue.cc:237] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:08.028836 620 catalog_manager.cc:5495] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130). New cstate: current_term: 1 leader_uuid: "aa9c6aec080f4158b114e5d1ea58e928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } health_report { overall_health: HEALTHY } } }
I20260924 16:03:08.084683 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:08.079215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.16.233.130:42079)
0924 16:03:08.079405 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:08.079410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:08.079427 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:03:08.079526 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:08.079530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:08.079603 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:08.079724 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:03:08.080150 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.080867 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.082552 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.082959 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.083055 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:08.083537 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:03:08.084037 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:03:08.084144 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:08.084259 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 16:03:08.084326 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:03:08.093168 1042 raft_consensus.cc:1261] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 1 FOLLOWER]: Refusing update from remote peer aa9c6aec080f4158b114e5d1ea58e928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:03:08.093168 769 raft_consensus.cc:1261] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Refusing update from remote peer aa9c6aec080f4158b114e5d1ea58e928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:03:08.094367 1119 consensus_queue.cc:1035] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:03:08.094761 1114 consensus_queue.cc:1035] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:03:08.111584 1122 mvcc.cc:205] Tried to move back new op lower bound from 7332928668018262016 to 7332928667749183488. Current Snapshot: MvccSnapshot[applied={T|T < 7332928668018262016 or (T in {7332928668018262016})}]
I20260924 16:03:08.116868 1123 mvcc.cc:205] Tried to move back new op lower bound from 7332928668018262016 to 7332928667749183488. Current Snapshot: MvccSnapshot[applied={T|T < 7332928668018262016}]
I20260924 16:03:08.127604 619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:03:08.127980 619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:03:08.132974 619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Sending DeleteTablet for 3 replicas of tablet 25e6aaf0fc4f4fddb6211b15331a1ff0
I20260924 16:03:08.134583 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e6aaf0fc4f4fddb6211b15331a1ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:03:08 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:44082
I20260924 16:03:08.135200 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e6aaf0fc4f4fddb6211b15331a1ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:03:08 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:50620
I20260924 16:03:08.136653 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e6aaf0fc4f4fddb6211b15331a1ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:03:08 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:33416
I20260924 16:03:08.141786 1132 tablet_replica.cc:330] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: stopping tablet replica
I20260924 16:03:08.142343 1132 raft_consensus.cc:2229] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:03:08.142608 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 684
I20260924 16:03:08.143013 1133 tablet_replica.cc:330] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: stopping tablet replica
I20260924 16:03:08.142979 1132 raft_consensus.cc:2258] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:03:08.143451 1133 raft_consensus.cc:2229] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:03:08.143975 1133 raft_consensus.cc:2258] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:03:08.144616 1134 tablet_replica.cc:330] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: stopping tablet replica
I20260924 16:03:08.145033 1134 raft_consensus.cc:2229] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:03:08.145828 1134 raft_consensus.cc:2258] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:03:08.146860 1133 ts_tablet_manager.cc:1905] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:03:08.147207 1132 ts_tablet_manager.cc:1905] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:03:08.154726 1132 ts_tablet_manager.cc:1918] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:03:08.155052 1132 log.cc:1196] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/25e6aaf0fc4f4fddb6211b15331a1ff0
I20260924 16:03:08.155530 1134 ts_tablet_manager.cc:1905] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:03:08.155658 1132 ts_tablet_manager.cc:1939] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P 9bf1c88666ba4eed9b38debf51e5e3ca: Deleting consensus metadata
I20260924 16:03:08.156131 1133 ts_tablet_manager.cc:1918] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:03:08.156385 1133 log.cc:1196] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/25e6aaf0fc4f4fddb6211b15331a1ff0
I20260924 16:03:08.157035 1133 ts_tablet_manager.cc:1939] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P e72cbc613172471caed8e4279e1e7008: Deleting consensus metadata
I20260924 16:03:08.157932 605 catalog_manager.cc:4841] TS 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975): tablet 25e6aaf0fc4f4fddb6211b15331a1ff0 (table test-table [id=dac69b5a86044b668f70f9bc2fdfc8fe]) successfully deleted
I20260924 16:03:08.159121 604 catalog_manager.cc:4841] TS e72cbc613172471caed8e4279e1e7008 (127.16.233.129:46021): tablet 25e6aaf0fc4f4fddb6211b15331a1ff0 (table test-table [id=dac69b5a86044b668f70f9bc2fdfc8fe]) successfully deleted
Sep 24 16:03:08 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 16:03:08.165334 1134 ts_tablet_manager.cc:1918] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:03:08.165565 1134 log.cc:1196] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/25e6aaf0fc4f4fddb6211b15331a1ff0
I20260924 16:03:08.166102 1134 ts_tablet_manager.cc:1939] T 25e6aaf0fc4f4fddb6211b15331a1ff0 P aa9c6aec080f4158b114e5d1ea58e928: Deleting consensus metadata
I20260924 16:03:08.167874 607 catalog_manager.cc:4841] TS aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079): tablet 25e6aaf0fc4f4fddb6211b15331a1ff0 (table test-table [id=dac69b5a86044b668f70f9bc2fdfc8fe]) successfully deleted
I20260924 16:03:08.169312 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:08.143675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.16.233.129:46021)
0924 16:03:08.143882 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:08.143889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:08.144216 (+ 327us) server_negotiation.cc:407] Connection header received
0924 16:03:08.144517 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:08.144523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:08.144622 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:08.144704 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:08.148593 (+ 3889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.149834 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.152088 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.152804 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.152918 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:08.161187 (+ 8269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:08.161206 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:08.161230 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:08.161309 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:08.165511 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:08.166761 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:08.166772 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:08.166776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:08.166956 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:08.167510 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:08.167516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:08.167519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:08.168039 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:08.168357 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:08.168670 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 16:03:08.168918 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:03:08.170208 810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:08.262360 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 684
I20260924 16:03:08.281809 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 821
Sep 24 16:03:08 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:03:08.298799 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:08.282642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.16.233.130:42079)
0924 16:03:08.282825 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:08.282834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:08.282896 (+ 62us) server_negotiation.cc:407] Connection header received
0924 16:03:08.283277 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:08.283287 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:08.283375 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:08.283513 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:08.284816 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.285833 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.287482 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.287916 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.288056 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:08.292396 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:08.292412 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:08.292429 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:08.292473 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:08.295713 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:08.296541 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:08.296548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:08.296551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:08.296700 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:08.297116 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:08.297121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:08.297124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:08.297481 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:08.297867 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:08.298145 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 16:03:08.298414 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:03:08.299571 946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:08.414276 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 821
I20260924 16:03:08.433815 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 958
Sep 24 16:03:08 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 16:03:08.449738 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:08.434706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.16.233.131:34975)
0924 16:03:08.434931 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:08.434938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:08.435094 (+ 156us) server_negotiation.cc:407] Connection header received
0924 16:03:08.435319 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:08.435324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:08.435387 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:08.435519 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:08.437077 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.437814 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.439327 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:08.439723 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:08.439867 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:08.444461 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:08.444477 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:08.444495 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:08.444532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:08.447325 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:08.447982 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:08.448005 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:08.448007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:08.448143 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:08.448471 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:08.448474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:08.448476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:08.448812 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:08.449040 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:08.449323 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 16:03:08.449487 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:03:08.450498 1083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:08.539816 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 958
I20260924 16:03:08.557430 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 588
I20260924 16:03:08.558718 652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:08.673528 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 588
I20260924 16:03:28.696103 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36011
--webserver_interface=127.16.233.190
--webserver_port=41179
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:03:29.045392 1140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:29.045723 1140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:29.045910 1140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:29.053363 1140 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:03:29.053480 1140 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:03:29.053567 1140 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:03:29.053647 1140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:03:29.053730 1140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:29.053833 1140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:29.053918 1140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:03:29.054003 1140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:03:29.064361 1140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=41179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:29.065824 1140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:29.067287 1140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:29.091924 1147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:29.092018 1149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:29.092007 1146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:29.092337 1140 server_base.cc:1034] running on GCE node
I20260924 16:03:29.093603 1140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:29.095353 1140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:29.096589 1140 hybrid_clock.cc:648] HybridClock initialized: now 1790265809096507 us; error 73 us; skew 500 ppm
I20260924 16:03:29.097019 1140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:29 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265809, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:29.101011 1140 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:03:29.102960 1140 webserver.cc:460] Webserver started at http://127.16.233.190:41179/ using document root <none> and password file <none>
I20260924 16:03:29.103798 1140 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:29.103942 1140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:29.110414 1140 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 16:03:29.113744 1157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:29.115350 1140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:03:29.115581 1140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c"
format_stamp: "Formatted at 2026-09-24 16:03:05 on dist-test-slave-xn4h"
server_key: "de38c3302acb03cf6551baeb9762bc26"
server_key_iv: "6fdb84f56464dd5bc748f7570762030e"
server_key_version: "encryptionkey@0"
I20260924 16:03:29.116812 1140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:29.133026 1140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:29.137974 1140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:29.138362 1140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:29.158756 1140 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:36011
I20260924 16:03:29.158788 1208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:36011 every 8 connection(s)
I20260924 16:03:29.160560 1140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:03:29.161172 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 1140
I20260924 16:03:29.162240 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:46021
--local_ip_for_outbound_sockets=127.16.233.129
--tserver_master_addrs=127.16.233.190:36011
--webserver_port=36287
--webserver_interface=127.16.233.129
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:03:29.168598 1209 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:03:29.172494 1209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Bootstrap starting.
I20260924 16:03:29.193683 1209 log.cc:824] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:29.206096 1209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:03:29.206748 1209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Bootstrap complete.
I20260924 16:03:29.216355 1209 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:29.217311 1209 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0f1d5c8d9e4a2c95d91c1e4ea79b7c, State: Initialized, Role: FOLLOWER
I20260924 16:03:29.217993 1209 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:29.218271 1209 raft_consensus.cc:388] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:03:29.218466 1209 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:03:29.218659 1209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:03:29.223075 1209 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:29.223641 1209 leader_election.cc:302] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be0f1d5c8d9e4a2c95d91c1e4ea79b7c; no voters:
I20260924 16:03:29.224367 1209 leader_election.cc:288] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:03:29.224519 1213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:03:29.225775 1213 raft_consensus.cc:686] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 2 LEADER]: Becoming Leader. State: Replica: be0f1d5c8d9e4a2c95d91c1e4ea79b7c, State: Running, Role: LEADER
I20260924 16:03:29.226490 1213 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:29.226891 1209 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:03:29.229241 1215 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0f1d5c8d9e4a2c95d91c1e4ea79b7c. Latest consensus state: current_term: 2 leader_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } } }
I20260924 16:03:29.229760 1215 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: This master's current role is: LEADER
I20260924 16:03:29.230408 1214 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } } }
I20260924 16:03:29.230773 1214 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: This master's current role is: LEADER
I20260924 16:03:29.234781 1218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:03:29.245921 1218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:03:29.252007 1218 catalog_manager.cc:1259] Loaded cluster ID: f02556094ae84fada10704aac5581816
I20260924 16:03:29.252153 1218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:03:29.256698 1218 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:03:29.258615 1218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Loaded TSK: 0
I20260924 16:03:29.307348 1218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Generated new TSK 1
I20260924 16:03:29.307803 1218 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:03:29.546809 1211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:29.547138 1211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:29.547299 1211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:29.555130 1211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:29.555258 1211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:29.555430 1211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:03:29.565723 1211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:46021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=36287
--tserver_master_addrs=127.16.233.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:29.567157 1211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:29.568631 1211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:29.587276 1237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:29.587755 1238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:29.588269 1240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:29.588999 1211 server_base.cc:1034] running on GCE node
I20260924 16:03:29.589660 1211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:29.590812 1211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:29.592061 1211 hybrid_clock.cc:648] HybridClock initialized: now 1790265809592000 us; error 59 us; skew 500 ppm
I20260924 16:03:29.592409 1211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:29 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265809, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:29.595866 1211 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:03:29.597589 1211 webserver.cc:460] Webserver started at http://127.16.233.129:36287/ using document root <none> and password file <none>
I20260924 16:03:29.598392 1211 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:29.598543 1211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:29.604565 1211 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:03:29.607743 1247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:29.609743 1211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:03:29.609953 1211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "e72cbc613172471caed8e4279e1e7008"
format_stamp: "Formatted at 2026-09-24 16:03:06 on dist-test-slave-xn4h"
server_key: "9f6c68d691104e670f557805b7218c04"
server_key_iv: "e0de546aeab8c0f7dbe312e8e73c483a"
server_key_version: "encryptionkey@0"
I20260924 16:03:29.611694 1211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:29.627415 1211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:29.632076 1211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:29.632381 1211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:29.633455 1211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:29.635033 1211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:29.635133 1211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:29.635301 1211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:29.635396 1211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:29.669950 1211 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:46021
I20260924 16:03:29.670060 1362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:46021 every 8 connection(s)
I20260924 16:03:29.671655 1211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:03:29.679661 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 1211
I20260924 16:03:29.680330 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:42079
--local_ip_for_outbound_sockets=127.16.233.130
--tserver_master_addrs=127.16.233.190:36011
--webserver_port=41853
--webserver_interface=127.16.233.130
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:03:29 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265809, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:03:29.695623 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:29.674484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:47943 (local address 127.16.233.190:36011)
0924 16:03:29.675086 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:29.675099 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:29.675880 (+ 781us) server_negotiation.cc:407] Connection header received
0924 16:03:29.678079 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:29.678092 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:29.678505 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:29.678805 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:29.680323 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:29.681646 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:29.683265 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:29.683914 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:29.684168 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:29.687899 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:29.687907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:29.687922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:29.687967 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:29.690989 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:29.691770 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:29.691777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:29.691783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:29.691943 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:29.692408 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:29.692412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:29.692414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:29.692892 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:29.693111 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:29.693890 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 16:03:29.694298 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
I20260924 16:03:29.695621 1368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:29.674762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:47943)
0924 16:03:29.675696 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:29.675749 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:29.677580 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:29.678948 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:29.678955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:29.679451 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:29.680079 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:29.680099 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:29.681839 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:29.681843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:29.683118 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:29.683130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:29.684202 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:29.684209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:29.684483 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:29.685195 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:29.685218 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:29.687616 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:29.691139 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:29.691149 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:29.691164 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:29.691611 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:29.692074 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:29.692080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:29.692084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:29.692278 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:29.693051 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:29.693059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:29.693396 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 16:03:29.693670 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 16:03:29.694242 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":394,"threads_started":1}
I20260924 16:03:29.698807 1364 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36011
I20260924 16:03:29.699193 1364 heartbeater.cc:464] Registering TS with master...
I20260924 16:03:29.700194 1364 heartbeater.cc:511] Master 127.16.233.190:36011 requested a full tablet report, sending...
I20260924 16:03:29.702648 1174 ts_manager.cc:194] Registered new tserver with Master: e72cbc613172471caed8e4279e1e7008 (127.16.233.129:46021)
I20260924 16:03:29.704200 1174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:47943
W20260924 16:03:30.025032 1369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:30.025334 1369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:30.025488 1369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:30.033156 1369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:30.033289 1369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:30.033501 1369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:03:30.043884 1369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:42079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=41853
--tserver_master_addrs=127.16.233.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:30.045349 1369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:30.046766 1369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:30.060645 1375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:30.060765 1374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:30.060742 1377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:30.061291 1369 server_base.cc:1034] running on GCE node
I20260924 16:03:30.061898 1369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:30.062937 1369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:30.064159 1369 hybrid_clock.cc:648] HybridClock initialized: now 1790265810064102 us; error 42 us; skew 500 ppm
I20260924 16:03:30.064512 1369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:30 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265810, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:30.067898 1369 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:03:30.069615 1369 webserver.cc:460] Webserver started at http://127.16.233.130:41853/ using document root <none> and password file <none>
I20260924 16:03:30.070446 1369 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:30.070642 1369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:30.076671 1369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:03:30.079809 1384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:30.081403 1369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:03:30.081606 1369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "aa9c6aec080f4158b114e5d1ea58e928"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "b3206ee1344e2ec269b0157f0a81b1a1"
server_key_iv: "be5469ebb88ccfe342063322b87ef5c3"
server_key_version: "encryptionkey@0"
I20260924 16:03:30.082705 1369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:30.094394 1369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:30.098753 1369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:30.098996 1369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:30.099987 1369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:30.101341 1369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:30.101436 1369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:30.101511 1369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:30.101567 1369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:30.140489 1369 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:42079
I20260924 16:03:30.140600 1500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:42079 every 8 connection(s)
I20260924 16:03:30.142011 1369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:03:30.146030 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 1369
I20260924 16:03:30.146857 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:34975
--local_ip_for_outbound_sockets=127.16.233.131
--tserver_master_addrs=127.16.233.190:36011
--webserver_port=41369
--webserver_interface=127.16.233.131
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:03:30 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265810, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:03:30.170004 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.144828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:48453 (local address 127.16.233.190:36011)
0924 16:03:30.145057 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.145063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.146478 (+ 1415us) server_negotiation.cc:407] Connection header received
0924 16:03:30.149593 (+ 3115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.149598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.149680 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.149778 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:30.152251 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.153012 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.155780 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.156593 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.156764 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.161715 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:30.161727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:30.161733 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:30.161780 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:30.165621 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:30.166901 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:30.166907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:30.166911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:30.167082 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:30.167691 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:30.167696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:30.167699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:30.168075 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:30.168226 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.169289 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.169567 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:03:30.171800 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.145341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:48453)
0924 16:03:30.146258 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:30.146335 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:30.149176 (+ 2841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:30.150108 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:30.150123 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:30.151006 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:30.151920 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.151953 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.153212 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.153220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.155558 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.155580 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.156846 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.156851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.157076 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.157893 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:30.157926 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:30.161358 (+ 3432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:30.165865 (+ 4507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:30.165880 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:30.165903 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:30.166732 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:30.167231 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:30.167239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:30.167245 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:30.167557 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:30.168240 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:30.168253 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:30.168816 (+ 563us) client_negotiation.cc:769] Sending connection context
0924 16:03:30.169174 (+ 358us) client_negotiation.cc:240] Negotiation successful
0924 16:03:30.169933 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":323,"threads_started":1}
I20260924 16:03:30.173722 1501 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36011
I20260924 16:03:30.174110 1501 heartbeater.cc:464] Registering TS with master...
I20260924 16:03:30.175066 1501 heartbeater.cc:511] Master 127.16.233.190:36011 requested a full tablet report, sending...
I20260924 16:03:30.176993 1174 ts_manager.cc:194] Registered new tserver with Master: aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079)
I20260924 16:03:30.178226 1174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:48453
W20260924 16:03:30.493221 1504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:30.493523 1504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:30.493686 1504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:30.501236 1504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:30.501369 1504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:30.501554 1504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:03:30.512336 1504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:34975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=41369
--tserver_master_addrs=127.16.233.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:30.513715 1504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:30.515163 1504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:30.528847 1510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:30.528898 1512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:30.528942 1509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:30.529874 1504 server_base.cc:1034] running on GCE node
I20260924 16:03:30.530602 1504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:30.531610 1504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:30.532842 1504 hybrid_clock.cc:648] HybridClock initialized: now 1790265810532789 us; error 48 us; skew 500 ppm
I20260924 16:03:30.533176 1504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:30 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265810, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:30.542827 1504 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:03:30.544528 1504 webserver.cc:460] Webserver started at http://127.16.233.131:41369/ using document root <none> and password file <none>
I20260924 16:03:30.545296 1504 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:30.545477 1504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:30.551151 1504 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:03:30.553970 1520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:30.555480 1504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:03:30.555665 1504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "9bf1c88666ba4eed9b38debf51e5e3ca"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "fab4d3a6c12ea3091e02611f34a9f8c3"
server_key_iv: "eae949b742f96fc96beb3c04195e8912"
server_key_version: "encryptionkey@0"
I20260924 16:03:30.556725 1504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:30.604714 1504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:30.609443 1504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:30.609758 1504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:30.610977 1504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:30.612872 1504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:30.613005 1504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:30.613138 1504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:30.613238 1504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:30.647557 1504 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:34975
I20260924 16:03:30.647647 1634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:34975 every 8 connection(s)
I20260924 16:03:30.649070 1504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:03:30.657369 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 1504
Sep 24 16:03:30 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265810, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:03:30.671936 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.651623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:40751 (local address 127.16.233.190:36011)
0924 16:03:30.651800 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.651805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.652986 (+ 1181us) server_negotiation.cc:407] Connection header received
0924 16:03:30.655033 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.655039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.655142 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.655254 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:30.656770 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.657889 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.660340 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.660897 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.661018 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.665098 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:30.665102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:30.665105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:30.665133 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:30.668397 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:30.669177 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:30.669182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:30.669184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:30.669356 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:30.669887 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:30.669891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:30.669893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:30.670310 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:30.670511 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.671279 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.671521 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:03:30.673971 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.652027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:40751)
0924 16:03:30.652831 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:30.652867 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:30.654743 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:30.655495 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:30.655504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:30.656004 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:30.656589 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.656606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.658053 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.658058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.660146 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.660165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.661142 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.661151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.661491 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.662225 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:30.662248 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:30.664888 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:30.668558 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:30.668570 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:30.668592 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:30.669025 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:30.669524 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:30.669529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:30.669534 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:30.669735 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:30.670482 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:30.670490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:30.670849 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 16:03:30.671993 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0924 16:03:30.672599 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":306,"threads_started":1}
I20260924 16:03:30.675441 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.658390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.16.233.190:36011)
0924 16:03:30.658903 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.658910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.659233 (+ 323us) server_negotiation.cc:407] Connection header received
0924 16:03:30.659391 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.659394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.659454 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.659544 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:30.661435 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.662465 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.664189 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.664624 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.664716 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.667598 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:30.667606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:30.667614 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:30.667668 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:30.671093 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:30.671988 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:30.671993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:30.671996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:30.672126 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:30.673100 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:30.673106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:30.673109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:30.673470 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:30.673731 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.674564 (+ 833us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.675040 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":231,"threads_started":1}
I20260924 16:03:30.675850 1635 heartbeater.cc:347] Connected to a master server at 127.16.233.190:36011
I20260924 16:03:30.676195 1635 heartbeater.cc:464] Registering TS with master...
I20260924 16:03:30.677117 1635 heartbeater.cc:511] Master 127.16.233.190:36011 requested a full tablet report, sending...
I20260924 16:03:30.678633 1174 ts_manager.cc:194] Registered new tserver with Master: 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975)
I20260924 16:03:30.679428 1174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:40751
I20260924 16:03:30.690431 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:03:30.697403 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.691354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.16.233.190:36011)
0924 16:03:30.691581 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.691589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.692024 (+ 435us) server_negotiation.cc:407] Connection header received
0924 16:03:30.692289 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.692293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.692434 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.692609 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:03:30.693077 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.693803 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.695392 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.695825 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.695964 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.696469 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:03:30.696711 (+ 242us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 16:03:30.697092 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44684: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 16:03:30.697566 1638 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44684: token expired
W20260924 16:03:30.698097 1093 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.233.190:36011)
I20260924 16:03:30.707934 1364 heartbeater.cc:503] Master 127.16.233.190:36011 was elected leader, sending a full tablet report...
I20260924 16:03:30.711153 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.698614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.16.233.190:36011)
0924 16:03:30.698778 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.698782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.698798 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:03:30.698919 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.698923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.698980 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.699068 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:30.700414 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.701112 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.702443 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.702820 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.702908 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.705414 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:30.705419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:30.705423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:30.705455 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:30.708238 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:30.708999 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:30.709003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:30.709006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:30.709153 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:30.709611 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:30.709617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:30.709622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:30.709939 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:30.710202 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.710575 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.710814 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:03:30.714459 1174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:03:30.722834 1174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:03:30.760936 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.735008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:34975 (local address 127.0.0.1:46564)
0924 16:03:30.736006 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:30.736070 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:30.736295 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:30.753330 (+ 17035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:30.753343 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:30.753393 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:30.753844 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.753858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.756051 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.756057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.758116 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.758130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.759775 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.759781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.759992 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.760044 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 16:03:30.760193 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 16:03:30.760461 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":107,"thread_start_us":16210,"threads_started":1}
I20260924 16:03:30.761858 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.735327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.16.233.131:34975)
0924 16:03:30.752151 (+ 16824us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.752166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.752221 (+ 55us) server_negotiation.cc:407] Connection header received
0924 16:03:30.752370 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.752384 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.752936 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.754075 (+ 1139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:30.754447 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.755892 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.758345 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.759552 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.759752 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.760023 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.761228 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.761467 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16547,"thread_start_us":17161,"threads_started":1}
I20260924 16:03:30.762487 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.738062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:42079 (local address 127.0.0.1:51248)
0924 16:03:30.752217 (+ 14155us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:30.752240 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:30.752379 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:30.754842 (+ 2463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:30.754849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:30.754881 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:30.755373 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.755386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.757000 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.757006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.759281 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.759292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.761432 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.761438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.761763 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.761803 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:03:30.761912 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 16:03:30.762066 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13996,"mutex_wait_us":52,"thread_start_us":12523,"threads_started":1}
I20260924 16:03:30.762915 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.735743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.16.233.129:46021)
0924 16:03:30.753349 (+ 17606us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.753360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.753393 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:03:30.753513 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.753521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.753766 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.754003 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:30.755029 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.756428 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.759204 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.760958 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.761258 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.761425 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.762449 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.762616 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17422,"thread_start_us":17272,"threads_started":1}
I20260924 16:03:30.763417 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.735197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:38496)
0924 16:03:30.751522 (+ 16325us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:30.751553 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:30.751743 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:30.754282 (+ 2539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:30.754287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:30.754345 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:30.754799 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.754814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.756599 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.756604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.759030 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.759045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.761629 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.761636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.761932 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.761965 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:03:30.762046 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 16:03:30.762202 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16085,"mutex_wait_us":98,"thread_start_us":700,"threads_started":1}
I20260924 16:03:30.766844 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.738154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.16.233.130:42079)
0924 16:03:30.753978 (+ 15824us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.753996 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.754025 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:03:30.754120 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.754128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.754382 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.754583 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:30.755567 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.756834 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.759425 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.760868 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.761058 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.761773 (+ 715us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.766450 (+ 4677us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.766605 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15371,"thread_start_us":15231,"threads_started":1}
I20260924 16:03:30.766173 1569 tablet_service.cc:1467] Processing CreateTablet for tablet fb49a90c036d456896286c54230e7056 (DEFAULT_TABLE table=test-table [id=c86fedb47b39452797d0b30525abde1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:03:30.766877 1296 tablet_service.cc:1467] Processing CreateTablet for tablet fb49a90c036d456896286c54230e7056 (DEFAULT_TABLE table=test-table [id=c86fedb47b39452797d0b30525abde1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:03:30.768082 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb49a90c036d456896286c54230e7056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:30.768965 1296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb49a90c036d456896286c54230e7056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:30.770203 1433 tablet_service.cc:1467] Processing CreateTablet for tablet fb49a90c036d456896286c54230e7056 (DEFAULT_TABLE table=test-table [id=c86fedb47b39452797d0b30525abde1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:03:30.771538 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb49a90c036d456896286c54230e7056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:30.781239 1651 tablet_bootstrap.cc:492] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Bootstrap starting.
I20260924 16:03:30.784512 1651 tablet_bootstrap.cc:654] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:30.785441 1652 tablet_bootstrap.cc:492] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: Bootstrap starting.
I20260924 16:03:30.785970 1651 log.cc:824] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:30.788933 1654 tablet_bootstrap.cc:492] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: Bootstrap starting.
I20260924 16:03:30.789268 1652 tablet_bootstrap.cc:654] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:30.789367 1651 tablet_bootstrap.cc:492] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: No bootstrap required, opened a new log
I20260924 16:03:30.789640 1651 ts_tablet_manager.cc:1397] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260924 16:03:30.790609 1652 log.cc:824] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:30.792415 1654 tablet_bootstrap.cc:654] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:30.793756 1654 log.cc:824] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:30.794011 1652 tablet_bootstrap.cc:492] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: No bootstrap required, opened a new log
I20260924 16:03:30.794341 1652 ts_tablet_manager.cc:1397] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 16:03:30.797075 1654 tablet_bootstrap.cc:492] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: No bootstrap required, opened a new log
I20260924 16:03:30.797367 1654 ts_tablet_manager.cc:1397] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260924 16:03:30.798089 1651 raft_consensus.cc:348] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:30.798596 1651 raft_consensus.cc:374] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:30.798715 1651 raft_consensus.cc:729] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72cbc613172471caed8e4279e1e7008, State: Initialized, Role: FOLLOWER
I20260924 16:03:30.799381 1651 consensus_queue.cc:260] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:30.805485 1652 raft_consensus.cc:348] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:30.806115 1652 raft_consensus.cc:374] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:30.806229 1652 raft_consensus.cc:729] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf1c88666ba4eed9b38debf51e5e3ca, State: Initialized, Role: FOLLOWER
I20260924 16:03:30.807003 1652 consensus_queue.cc:260] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:30.808171 1635 heartbeater.cc:503] Master 127.16.233.190:36011 was elected leader, sending a full tablet report...
I20260924 16:03:30.808240 1654 raft_consensus.cc:348] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:30.808691 1654 raft_consensus.cc:374] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:30.808804 1654 raft_consensus.cc:729] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa9c6aec080f4158b114e5d1ea58e928, State: Initialized, Role: FOLLOWER
I20260924 16:03:30.809424 1654 consensus_queue.cc:260] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:30.810261 1652 ts_tablet_manager.cc:1428] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260924 16:03:30.811105 1501 heartbeater.cc:503] Master 127.16.233.190:36011 was elected leader, sending a full tablet report...
I20260924 16:03:30.811296 1654 ts_tablet_manager.cc:1428] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260924 16:03:30.811795 1651 ts_tablet_manager.cc:1428] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Time spent starting tablet: real 0.022s user 0.008s sys 0.008s
I20260924 16:03:30.875962 1659 raft_consensus.cc:482] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:03:30.876327 1659 raft_consensus.cc:504] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:30.878113 1659 leader_election.cc:288] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975), e72cbc613172471caed8e4279e1e7008 (127.16.233.129:46021)
I20260924 16:03:30.887671 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.878763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:50241 (local address 127.16.233.129:46021)
0924 16:03:30.879036 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.879045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.879501 (+ 456us) server_negotiation.cc:407] Connection header received
0924 16:03:30.879738 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.879742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.879803 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.879942 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:30.880699 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.881878 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.885143 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.886480 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.886675 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.886756 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.887255 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.887403 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:03:30.887671 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.878566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:39813 (local address 127.16.233.131:34975)
0924 16:03:30.878792 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.878799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.879728 (+ 929us) server_negotiation.cc:407] Connection header received
0924 16:03:30.879987 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.879994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.880078 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.880203 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:30.880873 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.881931 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.885143 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.886480 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.886675 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.886756 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.887255 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.887398 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:03:30.887894 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.878626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:34975 (local address 127.16.233.130:39813)
0924 16:03:30.879608 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:30.879623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:30.879798 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:30.880241 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:30.880246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:30.880265 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:30.880697 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.880705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.882054 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.882060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.884899 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.884923 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.886781 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.886788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.887043 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.887073 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:03:30.887180 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:03:30.887389 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":224,"thread_start_us":628,"threads_started":1}
I20260924 16:03:30.887916 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.878415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.16.233.130:50241)
0924 16:03:30.879368 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:30.879387 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:30.879582 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:30.880018 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:30.880028 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:30.880054 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:30.880474 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.880488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.882056 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.882060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.884899 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:30.884921 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.886783 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.886788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:30.887024 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.887070 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 16:03:30.887179 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 16:03:30.887389 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":170,"thread_start_us":580,"threads_started":1}
I20260924 16:03:30.889249 1589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb49a90c036d456896286c54230e7056" candidate_uuid: "aa9c6aec080f4158b114e5d1ea58e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" is_pre_election: true
I20260924 16:03:30.889248 1316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb49a90c036d456896286c54230e7056" candidate_uuid: "aa9c6aec080f4158b114e5d1ea58e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72cbc613172471caed8e4279e1e7008" is_pre_election: true
I20260924 16:03:30.889856 1316 raft_consensus.cc:2454] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa9c6aec080f4158b114e5d1ea58e928 in term 0.
I20260924 16:03:30.889854 1589 raft_consensus.cc:2454] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa9c6aec080f4158b114e5d1ea58e928 in term 0.
I20260924 16:03:30.890655 1386 leader_election.cc:302] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf1c88666ba4eed9b38debf51e5e3ca, aa9c6aec080f4158b114e5d1ea58e928; no voters:
I20260924 16:03:30.891229 1659 raft_consensus.cc:2781] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:03:30.891398 1659 raft_consensus.cc:482] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:03:30.891512 1659 raft_consensus.cc:3037] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:30.894095 1659 raft_consensus.cc:504] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
W20260924 16:03:30.894441 1502 tablet.cc:2369] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:03:30.894986 1659 leader_election.cc:288] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [CANDIDATE]: Term 1 election: Requested vote from peers 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975), e72cbc613172471caed8e4279e1e7008 (127.16.233.129:46021)
I20260924 16:03:30.895586 1316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb49a90c036d456896286c54230e7056" candidate_uuid: "aa9c6aec080f4158b114e5d1ea58e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72cbc613172471caed8e4279e1e7008"
I20260924 16:03:30.895586 1589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb49a90c036d456896286c54230e7056" candidate_uuid: "aa9c6aec080f4158b114e5d1ea58e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca"
I20260924 16:03:30.895824 1589 raft_consensus.cc:3037] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:30.895824 1316 raft_consensus.cc:3037] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:30.899564 1589 raft_consensus.cc:2454] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa9c6aec080f4158b114e5d1ea58e928 in term 1.
I20260924 16:03:30.899564 1316 raft_consensus.cc:2454] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa9c6aec080f4158b114e5d1ea58e928 in term 1.
I20260924 16:03:30.900172 1385 leader_election.cc:302] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9bf1c88666ba4eed9b38debf51e5e3ca, aa9c6aec080f4158b114e5d1ea58e928, e72cbc613172471caed8e4279e1e7008; no voters:
I20260924 16:03:30.900666 1659 raft_consensus.cc:2781] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 16:03:30.900805 1636 tablet.cc:2369] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:03:30.901211 1659 raft_consensus.cc:686] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 LEADER]: Becoming Leader. State: Replica: aa9c6aec080f4158b114e5d1ea58e928, State: Running, Role: LEADER
I20260924 16:03:30.901830 1659 consensus_queue.cc:237] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:30.907137 1173 catalog_manager.cc:5495] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130). New cstate: current_term: 1 leader_uuid: "aa9c6aec080f4158b114e5d1ea58e928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } health_report { overall_health: HEALTHY } } }
W20260924 16:03:30.924041 1366 tablet.cc:2369] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:03:30.946544 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.940604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.16.233.130:42079)
0924 16:03:30.940805 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.940810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.940828 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:03:30.941071 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.941075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.941161 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.941289 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:03:30.941652 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.942492 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.944186 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:30.944635 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:30.944743 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:30.945329 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:03:30.945881 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:03:30.945995 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:30.946141 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 16:03:30.946217 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:03:30.955327 1316 raft_consensus.cc:1261] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Refusing update from remote peer aa9c6aec080f4158b114e5d1ea58e928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:03:30.955327 1589 raft_consensus.cc:1261] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 1 FOLLOWER]: Refusing update from remote peer aa9c6aec080f4158b114e5d1ea58e928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:03:30.956382 1662 consensus_queue.cc:1035] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:03:30.956807 1659 consensus_queue.cc:1035] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:03:30.987442 1172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:03:30.987732 1172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:03:30.991550 1172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Sending DeleteTablet for 3 replicas of tablet fb49a90c036d456896286c54230e7056
I20260924 16:03:30.992919 1296 tablet_service.cc:1514] Processing DeleteTablet for tablet fb49a90c036d456896286c54230e7056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:03:30 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:38496
I20260924 16:03:30.992910 1569 tablet_service.cc:1514] Processing DeleteTablet for tablet fb49a90c036d456896286c54230e7056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:03:30 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:46564
I20260924 16:03:30.993296 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 1211
I20260924 16:03:30.994021 1433 tablet_service.cc:1514] Processing DeleteTablet for tablet fb49a90c036d456896286c54230e7056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:03:30 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:51248
I20260924 16:03:31.016708 1675 tablet_replica.cc:330] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: stopping tablet replica
I20260924 16:03:31.017509 1675 raft_consensus.cc:2229] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:03:31.018388 1675 raft_consensus.cc:2258] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:03:31.019794 1673 tablet_replica.cc:330] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: stopping tablet replica
I20260924 16:03:31.020623 1674 tablet_replica.cc:330] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: stopping tablet replica
I20260924 16:03:31.021153 1673 raft_consensus.cc:2229] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:03:31.021347 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:30.995435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.16.233.129:46021)
0924 16:03:30.995690 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:30.995698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:30.995723 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:03:30.995802 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:30.995810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:30.995925 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:30.996067 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:30.998915 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:31.000620 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:31.003555 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:31.004533 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:31.004703 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:31.009818 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:31.009847 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:31.009877 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:31.009947 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:31.015603 (+ 5656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:31.017022 (+ 1419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:31.017037 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:31.017044 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:31.017366 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:31.018145 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:31.018153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:31.018159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:31.019596 (+ 1437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:31.020045 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:31.020680 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 16:03:31.020946 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 16:03:31.021924 1674 raft_consensus.cc:2229] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:03:31.022451 1358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:31.022550 1674 raft_consensus.cc:2258] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:03:31.023104 1673 raft_consensus.cc:2258] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:03:31.023730 1675 ts_tablet_manager.cc:1905] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:03:31.025276 1673 ts_tablet_manager.cc:1905] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:03:31.032970 1673 ts_tablet_manager.cc:1918] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:03:31.033195 1673 log.cc:1196] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/fb49a90c036d456896286c54230e7056
I20260924 16:03:31.033790 1673 ts_tablet_manager.cc:1939] T fb49a90c036d456896286c54230e7056 P 9bf1c88666ba4eed9b38debf51e5e3ca: Deleting consensus metadata
I20260924 16:03:31.036407 1159 catalog_manager.cc:4841] TS 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975): tablet fb49a90c036d456896286c54230e7056 (table test-table [id=c86fedb47b39452797d0b30525abde1e]) successfully deleted
I20260924 16:03:31.037092 1675 ts_tablet_manager.cc:1918] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:03:31.037343 1675 log.cc:1196] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/fb49a90c036d456896286c54230e7056
I20260924 16:03:31.038130 1675 ts_tablet_manager.cc:1939] T fb49a90c036d456896286c54230e7056 P aa9c6aec080f4158b114e5d1ea58e928: Deleting consensus metadata
I20260924 16:03:31.041245 1161 catalog_manager.cc:4841] TS aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079): tablet fb49a90c036d456896286c54230e7056 (table test-table [id=c86fedb47b39452797d0b30525abde1e]) successfully deleted
I20260924 16:03:31.190445 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 1211
W20260924 16:03:31.208977 1158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.129:46021 (error 108)
I20260924 16:03:31.209218 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 1369
I20260924 16:03:31.211123 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.210592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:38508)
0924 16:03:31.210809 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:31.210927 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 16:03:31.211447 1158 catalog_manager.cc:4584] TS e72cbc613172471caed8e4279e1e7008 (127.16.233.129:46021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb49a90c036d456896286c54230e7056: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46021: connect: Connection refused (error 111)
I20260924 16:03:31.226818 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.210126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.16.233.130:42079)
0924 16:03:31.210338 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:31.210344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:31.210482 (+ 138us) server_negotiation.cc:407] Connection header received
0924 16:03:31.210694 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:31.210700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:31.210785 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:31.210895 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:31.212629 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:31.213480 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:31.215625 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:31.216079 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:31.216179 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:31.219730 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:31.219746 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:31.219762 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:31.219804 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:31.223623 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:31.224564 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:31.224572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:31.224574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:31.224739 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:31.225164 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:31.225168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:31.225170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:31.225533 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:31.225805 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:31.226191 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 16:03:31.226464 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:03:31.227644 1496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:31.265146 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.264685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:38520)
0924 16:03:31.264898 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:31.264986 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 16:03:31.266287 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.265915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:38526)
0924 16:03:31.266081 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:31.266174 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 16:03:31.327277 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.326770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:38538)
0924 16:03:31.326974 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:31.327099 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 16:03:31.328483 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.328110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:38544)
0924 16:03:31.328303 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:31.328372 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:03:31.346258 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 1369
I20260924 16:03:31.365998 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 1504
I20260924 16:03:31.380244 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.366847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.16.233.131:34975)
0924 16:03:31.367043 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:31.367052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:31.367091 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:03:31.367292 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:31.367298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:31.367372 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:31.367482 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:31.368892 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:31.369652 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:31.371173 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:31.371586 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:31.371732 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:31.374413 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:31.374432 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:31.374450 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:31.374495 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:31.377111 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:31.377883 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:31.377893 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:31.377899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:31.378074 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:31.378535 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:31.378542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:31.378545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:31.378935 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:31.379270 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:31.379638 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 16:03:31.379862 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:03:31.381047 1630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:31.409068 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.408539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:38558)
0924 16:03:31.408779 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:31.408875 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 16:03:31.410281 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.409866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:38562)
0924 16:03:31.410102 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:31.410171 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 16:03:31.472987 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 1504
I20260924 16:03:31.490691 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 1140
I20260924 16:03:31.492079 1204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:03:31.608084 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:31.607452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46021 (local address 127.0.0.1:38568)
0924 16:03:31.607699 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:31.607819 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 16:03:31.608158 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 1140
I20260924 16:03:51.631033 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36011
--webserver_interface=127.16.233.190
--webserver_port=41179
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:36011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:03:51.970957 1682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:51.971323 1682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:51.971481 1682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:51.978866 1682 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:03:51.979048 1682 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:03:51.979147 1682 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:03:51.979246 1682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:03:51.979331 1682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:51.979439 1682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:51.979526 1682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:03:51.979607 1682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:03:51.989681 1682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:36011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:36011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=41179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:51.991151 1682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:51.992543 1682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:52.005344 1691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:52.005390 1688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:52.005422 1689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:52.005772 1682 server_base.cc:1034] running on GCE node
I20260924 16:03:52.006845 1682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:52.008406 1682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:52.009668 1682 hybrid_clock.cc:648] HybridClock initialized: now 1790265832009612 us; error 42 us; skew 500 ppm
I20260924 16:03:52.010042 1682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:52 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:52.029301 1682 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:03:52.031126 1682 webserver.cc:460] Webserver started at http://127.16.233.190:41179/ using document root <none> and password file <none>
I20260924 16:03:52.031916 1682 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:52.032071 1682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:52.037727 1682 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 16:03:52.040699 1698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:52.042066 1682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:03:52.042260 1682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c"
format_stamp: "Formatted at 2026-09-24 16:03:05 on dist-test-slave-xn4h"
server_key: "de38c3302acb03cf6551baeb9762bc26"
server_key_iv: "6fdb84f56464dd5bc748f7570762030e"
server_key_version: "encryptionkey@0"
I20260924 16:03:52.043264 1682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:52.081719 1682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:52.086109 1682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:52.086457 1682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:52.106035 1682 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:36011
I20260924 16:03:52.106070 1749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:36011 every 8 connection(s)
I20260924 16:03:52.107641 1682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:03:52.114719 1750 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:03:52.116235 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 1682
I20260924 16:03:52.117379 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:46021
--local_ip_for_outbound_sockets=127.16.233.129
--tserver_master_addrs=127.16.233.190:36011
--webserver_port=36287
--webserver_interface=127.16.233.129
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:03:52.119478 1750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Bootstrap starting.
I20260924 16:03:52.143484 1750 log.cc:824] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:52.160463 1750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:03:52.161095 1750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Bootstrap complete.
I20260924 16:03:52.170835 1750 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:52.171981 1750 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0f1d5c8d9e4a2c95d91c1e4ea79b7c, State: Initialized, Role: FOLLOWER
I20260924 16:03:52.172708 1750 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:52.172974 1750 raft_consensus.cc:388] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:03:52.173095 1750 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:03:52.173301 1750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:03:52.177580 1750 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:52.178169 1750 leader_election.cc:302] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be0f1d5c8d9e4a2c95d91c1e4ea79b7c; no voters:
I20260924 16:03:52.178911 1750 leader_election.cc:288] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:03:52.179081 1754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:03:52.180224 1754 raft_consensus.cc:686] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [term 3 LEADER]: Becoming Leader. State: Replica: be0f1d5c8d9e4a2c95d91c1e4ea79b7c, State: Running, Role: LEADER
I20260924 16:03:52.180857 1754 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } }
I20260924 16:03:52.181295 1750 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:03:52.183534 1756 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0f1d5c8d9e4a2c95d91c1e4ea79b7c. Latest consensus state: current_term: 3 leader_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } } }
I20260924 16:03:52.183923 1756 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: This master's current role is: LEADER
I20260924 16:03:52.183625 1755 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0f1d5c8d9e4a2c95d91c1e4ea79b7c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 36011 } } }
I20260924 16:03:52.184060 1755 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c [sys.catalog]: This master's current role is: LEADER
I20260924 16:03:52.185039 1760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:03:52.204960 1760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:03:52.206645 1760 catalog_manager.cc:1259] Loaded cluster ID: f02556094ae84fada10704aac5581816
I20260924 16:03:52.206851 1760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:03:52.211289 1760 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:03:52.213246 1760 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Loaded TSK: 0
I20260924 16:03:52.213500 1760 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be0f1d5c8d9e4a2c95d91c1e4ea79b7c: Loaded TSK: 1
F20260924 16:03:52.214944 1760 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb36e0cf9ad google::LogMessage::Fail() at ??:0
@ 0x7fb36e0d288a google::LogMessage::SendToLog() at ??:0
@ 0x7fb36e0cf4de google::LogMessage::Flush() at ??:0
@ 0x7fb36e0d0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb371e337d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb371e2273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb383afe1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb383afd783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb383b892a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb383b890c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb383bd35e9 std::function<>::operator()() at ??:0
@ 0x7fb383b01cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb383affc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb383b87610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb383b87439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb370d1b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb370d1e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb370d1e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb370cf9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb373fd36db start_thread at ??:0
@ 0x7fb36cf6961f clone at ??:0
W20260924 16:03:52.529162 1752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:52.529461 1752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:52.529559 1752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:52.537158 1752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:52.537289 1752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:52.537417 1752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:03:52.547434 1752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:46021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=36287
--tserver_master_addrs=127.16.233.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:52.549119 1752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:52.550429 1752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:52.563931 1798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:52.564036 1797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:52.564020 1800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:52.564774 1752 server_base.cc:1034] running on GCE node
I20260924 16:03:52.565402 1752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:52.566429 1752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:52.567657 1752 hybrid_clock.cc:648] HybridClock initialized: now 1790265832567615 us; error 50 us; skew 500 ppm
I20260924 16:03:52.567991 1752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:52 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:52.571151 1752 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:03:52.572700 1752 webserver.cc:460] Webserver started at http://127.16.233.129:36287/ using document root <none> and password file <none>
I20260924 16:03:52.573434 1752 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:52.573592 1752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:52.579299 1752 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:03:52.582233 1807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:52.583727 1752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:03:52.583914 1752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "e72cbc613172471caed8e4279e1e7008"
format_stamp: "Formatted at 2026-09-24 16:03:06 on dist-test-slave-xn4h"
server_key: "9f6c68d691104e670f557805b7218c04"
server_key_iv: "e0de546aeab8c0f7dbe312e8e73c483a"
server_key_version: "encryptionkey@0"
I20260924 16:03:52.584914 1752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:52.601305 1752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:52.605392 1752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:52.605702 1752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:52.606745 1752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:52.608605 1814 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 16:03:52.615890 1752 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:03:52.616067 1752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260924 16:03:52.616220 1752 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:03:52.619418 1752 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:03:52.619529 1752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260924 16:03:52.619899 1814 tablet_bootstrap.cc:492] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Bootstrap starting.
I20260924 16:03:52.628559 1814 log.cc:824] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:52.644635 1814 tablet_bootstrap.cc:492] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:03:52.645304 1814 tablet_bootstrap.cc:492] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Bootstrap complete.
I20260924 16:03:52.645776 1814 ts_tablet_manager.cc:1397] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.013s
I20260924 16:03:52.656561 1814 raft_consensus.cc:348] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:52.658258 1814 raft_consensus.cc:729] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72cbc613172471caed8e4279e1e7008, State: Initialized, Role: FOLLOWER
I20260924 16:03:52.659173 1814 consensus_queue.cc:260] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:52.660777 1814 ts_tablet_manager.cc:1428] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260924 16:03:52.664477 1752 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:46021
I20260924 16:03:52.664556 1924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:46021 every 8 connection(s)
I20260924 16:03:52.665769 1752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:03:52.666895 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 1752
I20260924 16:03:52.667728 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:42079
--local_ip_for_outbound_sockets=127.16.233.130
--tserver_master_addrs=127.16.233.190:36011
--webserver_port=41853
--webserver_interface=127.16.233.130
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:03:52.676277 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:52.668833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:52637)
0924 16:03:52.674718 (+ 5885us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:52.675020 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5520,"thread_start_us":392,"threads_started":1}
W20260924 16:03:52.677032 1925 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:36011 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.190:36011: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
I20260924 16:03:52.678025 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:52.677694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:47769)
0924 16:03:52.677850 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:52.677922 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 16:03:52.678925 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:52.678587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:33385)
0924 16:03:52.678732 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:52.678815 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:03:52.679212 1925 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:03:53.007797 1928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:53.008096 1928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:53.008191 1928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:53.015493 1928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:53.015617 1928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:53.015735 1928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:03:53.025604 1928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:42079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=41853
--tserver_master_addrs=127.16.233.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:53.026904 1928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:53.028239 1928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:53.041615 1933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:53.041591 1934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:53.042026 1936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:53.042456 1928 server_base.cc:1034] running on GCE node
I20260924 16:03:53.043107 1928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:53.044128 1928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:53.045334 1928 hybrid_clock.cc:648] HybridClock initialized: now 1790265833045286 us; error 37 us; skew 500 ppm
I20260924 16:03:53.045686 1928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:53 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:53.048839 1928 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:03:53.050256 1928 webserver.cc:460] Webserver started at http://127.16.233.130:41853/ using document root <none> and password file <none>
I20260924 16:03:53.050964 1928 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:53.051075 1928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:53.056475 1928 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 16:03:53.059192 1943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:53.060622 1928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:03:53.060818 1928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "aa9c6aec080f4158b114e5d1ea58e928"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "b3206ee1344e2ec269b0157f0a81b1a1"
server_key_iv: "be5469ebb88ccfe342063322b87ef5c3"
server_key_version: "encryptionkey@0"
I20260924 16:03:53.061853 1928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:53.085317 1928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:53.089681 1928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:53.089980 1928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:53.091090 1928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:53.092485 1928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:53.092597 1928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:53.092736 1928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:53.092823 1928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:53.126914 1928 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:42079
I20260924 16:03:53.126971 2057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:42079 every 8 connection(s)
I20260924 16:03:53.128477 1928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:03:53.133546 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:53.131622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:36335)
0924 16:03:53.132377 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:53.132636 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":289,"threads_started":1}
W20260924 16:03:53.134382 2058 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:36011 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.190:36011: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
I20260924 16:03:53.135231 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 1928
I20260924 16:03:53.135437 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:53.134965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:44475)
0924 16:03:53.135167 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:53.135270 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 16:03:53.136052 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:34975
--local_ip_for_outbound_sockets=127.16.233.131
--tserver_master_addrs=127.16.233.190:36011
--webserver_port=41369
--webserver_interface=127.16.233.131
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:03:53.136462 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:53.136063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:40907)
0924 16:03:53.136260 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:53.136350 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:03:53.136775 2058 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:03:53.482188 2061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:53.482573 2061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:53.482779 2061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:53.490396 2061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:53.490535 2061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:53.490732 2061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:03:53.501183 2061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:34975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=41369
--tserver_master_addrs=127.16.233.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:03:53.502635 2061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:53.504019 2061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:53.517735 2066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:53.517760 2069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:53.517812 2067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:53.518730 2061 server_base.cc:1034] running on GCE node
I20260924 16:03:53.519383 2061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:53.520453 2061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:53.521668 2061 hybrid_clock.cc:648] HybridClock initialized: now 1790265833521627 us; error 36 us; skew 500 ppm
I20260924 16:03:53.521970 2061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:03:53 dist-test-slave-xn4h krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:53.525332 2061 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:03:53.526928 2061 webserver.cc:460] Webserver started at http://127.16.233.131:41369/ using document root <none> and password file <none>
I20260924 16:03:53.527629 2061 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:53.527738 2061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:53.533501 2061 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:03:53.536588 2076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:53.538121 2061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:03:53.538362 2061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "9bf1c88666ba4eed9b38debf51e5e3ca"
format_stamp: "Formatted at 2026-09-24 16:03:07 on dist-test-slave-xn4h"
server_key: "fab4d3a6c12ea3091e02611f34a9f8c3"
server_key_iv: "eae949b742f96fc96beb3c04195e8912"
server_key_version: "encryptionkey@0"
I20260924 16:03:53.539419 2061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:53.561544 2061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:53.565860 2061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:53.566205 2061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:53.567267 2061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:53.568710 2061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:53.568821 2061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:53.568962 2061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:53.569052 2061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:53.603199 2061 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:34975
I20260924 16:03:53.603291 2190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:34975 every 8 connection(s)
I20260924 16:03:53.604755 2061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:03:53.609820 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:53.607898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:56501)
0924 16:03:53.608666 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:53.608913 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":281,"threads_started":1}
W20260924 16:03:53.610560 2191 heartbeater.cc:644] Failed to heartbeat to 127.16.233.190:36011 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.190:36011: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
I20260924 16:03:53.611418 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:53.611097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:60389)
0924 16:03:53.611240 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:53.611310 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 16:03:53.612210 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 2061
I20260924 16:03:53.612422 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:53.612062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:32805)
0924 16:03:53.612197 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:53.612270 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 16:03:53.612682 2191 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:03:53.680678 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:53.679912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:58847)
0924 16:03:53.680387 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:53.680479 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
I20260924 16:03:54.138589 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:54.137494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:43335)
0924 16:03:54.138162 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:54.138281 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":328,"threads_started":1}
I20260924 16:03:54.345820 2197 raft_consensus.cc:482] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:03:54.346185 2197 raft_consensus.cc:504] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:54.347848 2197 leader_election.cc:288] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975), aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079)
Sep 24 16:03:54 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
Sep 24 16:03:54 dist-test-slave-xn4h krb5kdc[575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:03:54.383176 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:54.348678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:44985 (local address 127.16.233.130:42079)
0924 16:03:54.361339 (+ 12661us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:54.361365 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:54.361904 (+ 539us) server_negotiation.cc:407] Connection header received
0924 16:03:54.363633 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:54.363641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:54.363982 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:54.364303 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:54.366321 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:54.368291 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:54.370120 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:54.370916 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:54.371216 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:54.375574 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:54.375581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:54.375594 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:54.375632 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:54.379154 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:54.380131 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:54.380138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:54.380144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:54.380298 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:54.380745 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:54.380752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:54.380755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:54.381305 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:54.381516 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:54.382388 (+ 872us) server_negotiation.cc:299] Negotiation successful
0924 16:03:54.382769 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12448,"thread_start_us":12312,"threads_started":1}
I20260924 16:03:54.383194 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:54.348291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:42079 (local address 127.16.233.129:44985)
0924 16:03:54.361751 (+ 13460us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:54.361770 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:54.361915 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:54.364328 (+ 2413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:54.364341 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:54.365227 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:54.366039 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:54.366061 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:54.368553 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:54.368557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:54.369971 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:54.369986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:54.371224 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:54.371231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:54.371511 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:54.372325 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:54.372357 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:54.375273 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:54.379309 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:54.379321 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:54.379323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:54.379978 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:54.380432 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:54.380438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:54.380443 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:54.380625 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:54.381461 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:54.381469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:54.381822 (+ 353us) client_negotiation.cc:769] Sending connection context
0924 16:03:54.382115 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 16:03:54.382710 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13316,"mutex_wait_us":208,"spinlock_wait_cycles":39040,"thread_start_us":13313,"threads_started":1}
I20260924 16:03:54.383299 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:54.348211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:34975 (local address 127.16.233.129:52897)
0924 16:03:54.360784 (+ 12573us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:54.360836 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:54.361575 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:54.364331 (+ 2756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:54.364341 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:54.365223 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:54.366039 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:54.366061 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:54.368500 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:54.368505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:54.369963 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:54.369980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:54.371152 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:54.371159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:54.371505 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:54.372394 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:54.372411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:54.375113 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:54.379282 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:54.379293 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:54.379316 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:54.379937 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:54.380404 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:54.380409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:54.380413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:54.380626 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:54.381459 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:54.381467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:54.381822 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 16:03:54.382115 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 16:03:54.382866 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12426,"mutex_wait_us":378,"spinlock_wait_cycles":23936,"thread_start_us":12263,"threads_started":1}
I20260924 16:03:54.383199 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:54.348359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:52897 (local address 127.16.233.131:34975)
0924 16:03:54.360375 (+ 12016us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:54.360400 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:54.361470 (+ 1070us) server_negotiation.cc:407] Connection header received
0924 16:03:54.363362 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:54.363372 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:54.363813 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:54.364189 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:54.366264 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:54.368291 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:54.370125 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:54.370911 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:54.371176 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:54.375451 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:54.375460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:54.375476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:54.375520 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:54.379106 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:54.380095 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:54.380103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:54.380109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:54.380275 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:54.380743 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:54.380749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:54.380753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:54.381307 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:54.381516 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:54.382372 (+ 856us) server_negotiation.cc:299] Negotiation successful
0924 16:03:54.382764 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11826,"thread_start_us":11673,"threads_started":1}
W20260924 16:03:54.386897 1811 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
W20260924 16:03:54.387229 1809 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
I20260924 16:03:54.387389 1809 leader_election.cc:302] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72cbc613172471caed8e4279e1e7008; no voters: 9bf1c88666ba4eed9b38debf51e5e3ca, aa9c6aec080f4158b114e5d1ea58e928
I20260924 16:03:54.387817 2197 raft_consensus.cc:2726] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:03:54.614205 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:54.613244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:38581)
0924 16:03:54.613771 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:54.613946 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":245,"threads_started":1}
I20260924 16:03:54.682199 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:54.681747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:41243)
0924 16:03:54.681939 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:54.682058 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 16:03:55.140373 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:55.139511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:47531)
0924 16:03:55.140094 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:55.140205 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":241,"threads_started":1}
I20260924 16:03:55.615885 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:55.615210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:58611)
0924 16:03:55.615645 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:55.615732 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":235,"threads_started":1}
I20260924 16:03:55.684084 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:55.683289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:42145)
0924 16:03:55.683806 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:55.683897 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
I20260924 16:03:56.142504 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:56.141537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:42669)
0924 16:03:56.142098 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:56.142209 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":283,"threads_started":1}
I20260924 16:03:56.175837 2207 raft_consensus.cc:482] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:03:56.176090 2207 raft_consensus.cc:504] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:56.176833 2207 leader_election.cc:288] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975), aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079)
W20260924 16:03:56.177965 1809 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
W20260924 16:03:56.178165 1811 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
I20260924 16:03:56.178385 1811 leader_election.cc:302] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72cbc613172471caed8e4279e1e7008; no voters: 9bf1c88666ba4eed9b38debf51e5e3ca, aa9c6aec080f4158b114e5d1ea58e928
I20260924 16:03:56.178642 2207 raft_consensus.cc:2726] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:03:56.617600 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:56.616901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:51041)
0924 16:03:56.617349 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:56.617427 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260924 16:03:56.685698 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:56.685002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:44555)
0924 16:03:56.685438 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:56.685533 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":213,"threads_started":1}
I20260924 16:03:57.144510 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:57.143607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:34835)
0924 16:03:57.144186 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:57.144312 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":233,"threads_started":1}
I20260924 16:03:57.619354 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:57.618633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:38049)
0924 16:03:57.619095 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:57.619176 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":226,"threads_started":1}
I20260924 16:03:57.687522 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:57.686710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:32813)
0924 16:03:57.687222 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:57.687327 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":236,"threads_started":1}
I20260924 16:03:58.047277 2213 raft_consensus.cc:482] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:03:58.047479 2213 raft_consensus.cc:504] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:03:58.048151 2213 leader_election.cc:288] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975), aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079)
W20260924 16:03:58.049203 1811 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
W20260924 16:03:58.049398 1809 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
I20260924 16:03:58.049533 1809 leader_election.cc:302] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72cbc613172471caed8e4279e1e7008; no voters: 9bf1c88666ba4eed9b38debf51e5e3ca, aa9c6aec080f4158b114e5d1ea58e928
I20260924 16:03:58.049808 2213 raft_consensus.cc:2726] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:03:58.146486 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:58.145547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:48151)
0924 16:03:58.146134 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:58.146223 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":239,"threads_started":1}
I20260924 16:03:58.620944 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:58.620270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:43201)
0924 16:03:58.620696 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:58.620782 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260924 16:03:58.689198 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:58.688401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:57783)
0924 16:03:58.688920 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:58.689034 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":291,"threads_started":1}
I20260924 16:03:59.148424 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:59.147467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:59149)
0924 16:03:59.148027 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:59.148175 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":257,"threads_started":1}
I20260924 16:03:59.622836 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:59.621991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:36779)
0924 16:03:59.622564 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:59.622653 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":325,"threads_started":1}
I20260924 16:03:59.690990 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:59.690121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:37015)
0924 16:03:59.690624 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:59.690723 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":288,"threads_started":1}
I20260924 16:04:00.150410 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:00.149481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:48277)
0924 16:04:00.150035 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:00.150133 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":249,"threads_started":1}
I20260924 16:04:00.624723 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:00.623922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:50711)
0924 16:04:00.624443 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:00.624535 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":227,"threads_started":1}
I20260924 16:04:00.692723 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:00.692004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:51761)
0924 16:04:00.692469 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:00.692560 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
I20260924 16:04:01.152145 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:01.151384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:57453)
0924 16:04:01.151812 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:01.151912 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":226,"threads_started":1}
I20260924 16:04:01.626631 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:01.625831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:54025)
0924 16:04:01.626295 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:01.626434 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
I20260924 16:04:01.647330 2227 raft_consensus.cc:482] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:01.647555 2227 raft_consensus.cc:504] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:04:01.648172 2227 leader_election.cc:288] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975), aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079)
W20260924 16:04:01.649248 1811 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
W20260924 16:04:01.649430 1809 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
I20260924 16:04:01.649565 1809 leader_election.cc:302] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72cbc613172471caed8e4279e1e7008; no voters: 9bf1c88666ba4eed9b38debf51e5e3ca, aa9c6aec080f4158b114e5d1ea58e928
I20260924 16:04:01.649837 2227 raft_consensus.cc:2726] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:04:01.694797 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:01.693792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:33529)
0924 16:04:01.694474 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:01.694606 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":343,"threads_started":1}
I20260924 16:04:02.154050 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:02.153282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:54363)
0924 16:04:02.153779 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:02.153890 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":234,"threads_started":1}
I20260924 16:04:02.628489 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:02.627698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:43909)
0924 16:04:02.628208 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:02.628321 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":230,"threads_started":1}
I20260924 16:04:02.696671 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:02.695941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:34949)
0924 16:04:02.696412 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:02.696506 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":255,"threads_started":1}
I20260924 16:04:03.156067 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:03.155245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:46377)
0924 16:04:03.155760 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:03.155858 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":281,"threads_started":1}
I20260924 16:04:03.630491 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:03.629628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:51949)
0924 16:04:03.630134 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:03.630235 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":303,"threads_started":1}
I20260924 16:04:03.698529 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:03.697680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:41743)
0924 16:04:03.698174 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:03.698266 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":255,"threads_started":1}
I20260924 16:04:04.157856 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:04.157067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:46617)
0924 16:04:04.157596 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:04.157689 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":252,"threads_started":1}
I20260924 16:04:04.632262 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:04.631459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:51785)
0924 16:04:04.631984 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:04.632098 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":264,"threads_started":1}
I20260924 16:04:04.697053 2237 raft_consensus.cc:482] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:04.697261 2237 raft_consensus.cc:504] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf1c88666ba4eed9b38debf51e5e3ca" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 34975 } } peers { permanent_uuid: "e72cbc613172471caed8e4279e1e7008" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46021 } } peers { permanent_uuid: "aa9c6aec080f4158b114e5d1ea58e928" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 42079 } }
I20260924 16:04:04.697907 2237 leader_election.cc:288] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975), aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079)
W20260924 16:04:04.698957 1809 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bf1c88666ba4eed9b38debf51e5e3ca (127.16.233.131:34975): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
W20260924 16:04:04.699142 1811 leader_election.cc:341] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa9c6aec080f4158b114e5d1ea58e928 (127.16.233.130:42079): Not found: Tablet not found: fb49a90c036d456896286c54230e7056
I20260924 16:04:04.699278 1811 leader_election.cc:302] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72cbc613172471caed8e4279e1e7008; no voters: 9bf1c88666ba4eed9b38debf51e5e3ca, aa9c6aec080f4158b114e5d1ea58e928
I20260924 16:04:04.699529 2237 raft_consensus.cc:2726] T fb49a90c036d456896286c54230e7056 P e72cbc613172471caed8e4279e1e7008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:04:04.700265 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:04.699400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:35879)
0924 16:04:04.699939 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:04.700042 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":273,"threads_started":1}
I20260924 16:04:05.159706 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:05.158927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:33265)
0924 16:04:05.159421 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:05.159516 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":256,"threads_started":1}
I20260924 16:04:05.634212 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:05.633457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:58031)
0924 16:04:05.633930 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:05.634037 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":242,"threads_started":1}
I20260924 16:04:05.701902 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:05.701154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:34493)
0924 16:04:05.701632 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:05.701724 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":267,"threads_started":1}
I20260924 16:04:06.161386 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:06.160690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:37821)
0924 16:04:06.161127 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:06.161227 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":214,"threads_started":1}
I20260924 16:04:06.636018 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:06.635229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:40257)
0924 16:04:06.635738 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:06.635846 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":262,"threads_started":1}
I20260924 16:04:06.703529 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:06.702820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:34955)
0924 16:04:06.703261 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:06.703356 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":229,"threads_started":1}
I20260924 16:04:07.163153 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:07.162364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:52917)
0924 16:04:07.162842 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:07.162947 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":240,"threads_started":1}
I20260924 16:04:07.637882 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:07.637021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:39129)
0924 16:04:07.637574 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:07.637678 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":287,"threads_started":1}
I20260924 16:04:07.705574 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:07.704783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:59193)
0924 16:04:07.705292 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:07.705396 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":253,"threads_started":1}
I20260924 16:04:08.165475 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:08.164626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.130:60461)
0924 16:04:08.165190 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:08.165305 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":233,"threads_started":1}
I20260924 16:04:08.623211 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 1752
I20260924 16:04:08.638689 2250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:08.624395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.16.233.129:46021)
0924 16:04:08.624975 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:08.624987 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:08.625034 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:04:08.625113 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:08.625120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:08.625380 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:08.625561 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:08.627228 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:08.628218 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:08.629505 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:08.630091 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:08.630245 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:08.632922 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:08.632935 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:08.632950 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:08.632988 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:08.635720 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:08.636462 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:08.636470 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:08.636473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:08.636619 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:08.636998 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:08.637007 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:08.637009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:08.637390 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:08.637712 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:08.638046 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 16:04:08.638357 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260924 16:04:08.640326 1919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:08.640410 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:08.639413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.131:56153)
0924 16:04:08.640035 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:08.640187 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260924 16:04:08.731752 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 1752
I20260924 16:04:08.731814 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:08.730673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:36011 (local address 127.16.233.129:59769)
0924 16:04:08.731285 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:08.731500 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.190:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":321,"threads_started":1}
I20260924 16:04:08.750837 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 1928
I20260924 16:04:08.764357 2254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:08.751663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.16.233.130:42079)
0924 16:04:08.752137 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:08.752141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:08.752156 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:04:08.752209 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:08.752212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:08.752267 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:08.752382 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:08.753770 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:08.754608 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:08.755846 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:08.756288 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:08.756418 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:08.759023 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:08.759029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:08.759032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:08.759065 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:08.761541 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:08.762251 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:08.762258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:08.762262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:08.762469 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:08.762840 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:08.762847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:08.762850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:08.763145 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:08.763424 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:08.763729 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 16:04:08.763973 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":228,"threads_started":1}
I20260924 16:04:08.765117 2053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:08.850090 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 1928
I20260924 16:04:08.867952 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 2061
I20260924 16:04:08.881283 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:08.868735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.16.233.131:34975)
0924 16:04:08.869350 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:08.869355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:08.869372 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:04:08.869431 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:08.869435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:08.869507 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:08.869601 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:08.870994 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:08.871770 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:08.872888 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:08.873328 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:08.873418 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:08.875973 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:08.875979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:08.875982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:08.876021 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:08.878934 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:08.879626 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:08.879629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:08.879631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:08.879784 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:08.880110 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:08.880113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:08.880115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:08.880372 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:08.880597 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:08.880790 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 16:04:08.880977 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":346,"threads_started":1}
I20260924 16:04:08.881986 2186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:08.967809 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 2061
2026-09-24T16:04:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64913 ms)
[----------] 30 tests from SecurityITest (303170 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:09 dist-test-slave-xn4h krb5kdc[2260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:09 dist-test-slave-xn4h krb5kdc[2260](info): set up 2 sockets
Sep 24 16:04:09 dist-test-slave-xn4h krb5kdc[2260](info): commencing operation
krb5kdc: starting...
W20260924 16:04:10.104334 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:10 dist-test-slave-xn4h krb5kdc[2260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:04:10.365778 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:33233
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:33233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:10.705106 2273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:10.705384 2273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:10.705551 2273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:10.713006 2273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:10.713115 2273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:10.713225 2273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:10.713320 2273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:10.713403 2273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:10.723371 2273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:33233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:33233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:10.724684 2273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:10.725998 2273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:10.738148 2278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:10.738067 2279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:10.738108 2281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:10.738855 2273 server_base.cc:1034] running on GCE node
I20260924 16:04:10.739651 2273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:10.741122 2273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:10.742416 2273 hybrid_clock.cc:648] HybridClock initialized: now 1790265850742375 us; error 47 us; skew 500 ppm
I20260924 16:04:10.742765 2273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:10 dist-test-slave-xn4h krb5kdc[2260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265850, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:10.751190 2273 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:04:10.752772 2273 webserver.cc:460] Webserver started at http://127.16.233.190:36655/ using document root <none> and password file <none>
I20260924 16:04:10.753471 2273 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:10.753615 2273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:10.753925 2273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:10.756920 2273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "e517c1875ddf483baf63f84b93392c68"
format_stamp: "Formatted at 2026-09-24 16:04:10 on dist-test-slave-xn4h"
server_key: "6c7f02f28b8b4b9e08d8b1e1dfc4505f"
server_key_iv: "372d4dd15e8b8481d92dd390d8c8a149"
server_key_version: "encryptionkey@0"
I20260924 16:04:10.757671 2273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "e517c1875ddf483baf63f84b93392c68"
format_stamp: "Formatted at 2026-09-24 16:04:10 on dist-test-slave-xn4h"
server_key: "6c7f02f28b8b4b9e08d8b1e1dfc4505f"
server_key_iv: "372d4dd15e8b8481d92dd390d8c8a149"
server_key_version: "encryptionkey@0"
I20260924 16:04:10.762898 2273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:04:10.766191 2288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:10.767482 2273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:10.767652 2273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "e517c1875ddf483baf63f84b93392c68"
format_stamp: "Formatted at 2026-09-24 16:04:10 on dist-test-slave-xn4h"
server_key: "6c7f02f28b8b4b9e08d8b1e1dfc4505f"
server_key_iv: "372d4dd15e8b8481d92dd390d8c8a149"
server_key_version: "encryptionkey@0"
I20260924 16:04:10.767841 2273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:10.794232 2273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:10.798537 2273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:10.798789 2273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:10.817898 2273 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:33233
I20260924 16:04:10.817929 2343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:33233 every 8 connection(s)
I20260924 16:04:10.819334 2273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:10.820966 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 2273
I20260924 16:04:10.821210 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:10.822059 17318 external_mini_cluster.cc:1428] Setting key 465528d8a1a161b422f29bcbf5ee7a75
I20260924 16:04:10.824149 2344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:04:10 dist-test-slave-xn4h krb5kdc[2260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:10.838032 2344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68: Bootstrap starting.
I20260924 16:04:10.841249 2344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:10.842247 2344 log.cc:824] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:10.845115 2344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68: No bootstrap required, opened a new log
I20260924 16:04:10.845407 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:10.824154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.16.233.190:33233)
0924 16:04:10.825003 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:10.825014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:10.825055 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:04:10.826767 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:10.826790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:10.827153 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:10.827491 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:10.829310 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:10.830697 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:10.832623 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:10.833294 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:10.833427 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:10.838186 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:10.838197 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:10.838215 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:10.838261 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:10.841301 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:10.842035 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:10.842042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:10.842048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:10.842214 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:10.842678 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:10.842683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:10.842688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:10.843236 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:10.843554 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:10.844009 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 16:04:10.844418 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":385,"threads_started":1}
I20260924 16:04:10.853212 2344 raft_consensus.cc:348] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e517c1875ddf483baf63f84b93392c68" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33233 } }
I20260924 16:04:10.853596 2344 raft_consensus.cc:374] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:10.853704 2344 raft_consensus.cc:729] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517c1875ddf483baf63f84b93392c68, State: Initialized, Role: FOLLOWER
I20260924 16:04:10.854239 2344 consensus_queue.cc:260] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e517c1875ddf483baf63f84b93392c68" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33233 } }
I20260924 16:04:10.854528 2344 raft_consensus.cc:388] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:10.854669 2344 raft_consensus.cc:482] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:10.854838 2344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:10.857053 2344 raft_consensus.cc:504] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e517c1875ddf483baf63f84b93392c68" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33233 } }
I20260924 16:04:10.857637 2344 leader_election.cc:302] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e517c1875ddf483baf63f84b93392c68; no voters:
I20260924 16:04:10.858209 2344 leader_election.cc:288] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:10.858372 2349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:10.859089 2349 raft_consensus.cc:686] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [term 1 LEADER]: Becoming Leader. State: Replica: e517c1875ddf483baf63f84b93392c68, State: Running, Role: LEADER
I20260924 16:04:10.859697 2349 consensus_queue.cc:237] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e517c1875ddf483baf63f84b93392c68" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33233 } }
I20260924 16:04:10.860284 2344 sys_catalog.cc:564] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:10.862326 2351 sys_catalog.cc:455] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e517c1875ddf483baf63f84b93392c68. Latest consensus state: current_term: 1 leader_uuid: "e517c1875ddf483baf63f84b93392c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e517c1875ddf483baf63f84b93392c68" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33233 } } }
I20260924 16:04:10.862720 2351 sys_catalog.cc:458] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:10.863248 2350 sys_catalog.cc:455] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e517c1875ddf483baf63f84b93392c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e517c1875ddf483baf63f84b93392c68" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33233 } } }
I20260924 16:04:10.863569 2350 sys_catalog.cc:458] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:10.864135 2356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:10.870633 2356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:10.879724 2356 catalog_manager.cc:1347] Generated new cluster ID: 09f17f31aba44eef8c65114715b3b0c9
I20260924 16:04:10.879871 2356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:10.890574 2356 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:10.891880 2356 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:10.904572 2356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68: Generated new TSK 0
I20260924 16:04:10.905607 2356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 16:04:10.969659 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:33233
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:11.301600 2370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:11.301913 2370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:11.302060 2370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:11.309623 2370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:11.309731 2370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:11.309854 2370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:04:11.319708 2370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:33233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:11.320971 2370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:11.322208 2370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:11.335503 2375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:11.335526 2376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:11.335557 2378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:11.336649 2370 server_base.cc:1034] running on GCE node
I20260924 16:04:11.337248 2370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:11.338201 2370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:11.339428 2370 hybrid_clock.cc:648] HybridClock initialized: now 1790265851339396 us; error 75 us; skew 500 ppm
I20260924 16:04:11.339792 2370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:11 dist-test-slave-xn4h krb5kdc[2260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265851, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:11.343039 2370 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:04:11.344571 2370 webserver.cc:460] Webserver started at http://127.16.233.129:42277/ using document root <none> and password file <none>
I20260924 16:04:11.345309 2370 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:11.345408 2370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:11.345665 2370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:11.348574 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "950c55210d1f4ec8bf99e04e3d0357b2"
format_stamp: "Formatted at 2026-09-24 16:04:11 on dist-test-slave-xn4h"
server_key: "478051f390cfcba4b688ae51b333d36c"
server_key_iv: "02e2661d89a2ccf1844ac9d029d2ed06"
server_key_version: "encryptionkey@0"
I20260924 16:04:11.349289 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "950c55210d1f4ec8bf99e04e3d0357b2"
format_stamp: "Formatted at 2026-09-24 16:04:11 on dist-test-slave-xn4h"
server_key: "478051f390cfcba4b688ae51b333d36c"
server_key_iv: "02e2661d89a2ccf1844ac9d029d2ed06"
server_key_version: "encryptionkey@0"
I20260924 16:04:11.354514 2370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:04:11.357924 2385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:11.359287 2370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:11.359426 2370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "950c55210d1f4ec8bf99e04e3d0357b2"
format_stamp: "Formatted at 2026-09-24 16:04:11 on dist-test-slave-xn4h"
server_key: "478051f390cfcba4b688ae51b333d36c"
server_key_iv: "02e2661d89a2ccf1844ac9d029d2ed06"
server_key_version: "encryptionkey@0"
I20260924 16:04:11.359599 2370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:11.379138 2370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:11.383502 2370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:11.383751 2370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:11.384670 2370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:11.385977 2370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:11.386049 2370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:11.386134 2370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:11.386179 2370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:11.419356 2370 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:44515
I20260924 16:04:11.419404 2502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:44515 every 8 connection(s)
I20260924 16:04:11.420667 2370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:11.423756 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 2370
I20260924 16:04:11.423985 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:04:11.424624 17318 external_mini_cluster.cc:1428] Setting key 6daa7bd9bae5e18e9ca2847b9919f946
Sep 24 16:04:11 dist-test-slave-xn4h krb5kdc[2260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265851, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:11.442757 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:11.423194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:41613 (local address 127.16.233.190:33233)
0924 16:04:11.423679 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:11.423686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:11.424831 (+ 1145us) server_negotiation.cc:407] Connection header received
0924 16:04:11.427205 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:11.427211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:11.427304 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:11.427420 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:11.428846 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:11.429985 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:11.431573 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:11.432319 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:11.432475 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:11.435878 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:11.435886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:11.435894 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:11.435939 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:11.439464 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:11.440183 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:11.440191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:11.440198 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:11.440343 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:11.440748 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:11.440755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:11.440761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:11.441138 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:11.441302 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:11.441968 (+ 666us) server_negotiation.cc:299] Negotiation successful
0924 16:04:11.442231 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260924 16:04:11.443473 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:11.423785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:33233 (local address 127.16.233.129:41613)
0924 16:04:11.424650 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:11.424697 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:11.426939 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:11.427622 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:11.427631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:11.428133 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:11.428670 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:11.428687 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:11.430125 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:11.430129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:11.431423 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:11.431435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:11.432521 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:11.432525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:11.432713 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:11.433391 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:11.433417 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:11.435679 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:11.439609 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:11.439615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:11.439628 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:11.440052 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:11.440452 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:11.440455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:11.440457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:11.440623 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:11.441254 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:11.441259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:11.441589 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 16:04:11.441812 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 16:04:11.442182 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":349,"threads_started":1}
I20260924 16:04:11.445151 2503 heartbeater.cc:347] Connected to a master server at 127.16.233.190:33233
I20260924 16:04:11.445465 2503 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:11.446175 2503 heartbeater.cc:511] Master 127.16.233.190:33233 requested a full tablet report, sending...
I20260924 16:04:11.448724 2307 ts_manager.cc:194] Registered new tserver with Master: 950c55210d1f4ec8bf99e04e3d0357b2 (127.16.233.129:44515)
I20260924 16:04:11.450928 2307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:41613
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 16:04:11.482726 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:33233
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:11.809434 2509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:11.809718 2509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:11.809813 2509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:11.817015 2509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:11.817183 2509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:11.817373 2509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:04:11.827180 2509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:33233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:11.828454 2509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:11.829799 2509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:11.842382 2514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:11.842418 2517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:11.842474 2515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:11.843006 2509 server_base.cc:1034] running on GCE node
I20260924 16:04:11.843667 2509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:11.844622 2509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:11.845840 2509 hybrid_clock.cc:648] HybridClock initialized: now 1790265851845806 us; error 31 us; skew 500 ppm
I20260924 16:04:11.846155 2509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:11 dist-test-slave-xn4h krb5kdc[2260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265851, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:11.849225 2509 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:04:11.850708 2509 webserver.cc:460] Webserver started at http://127.16.233.130:37941/ using document root <none> and password file <none>
I20260924 16:04:11.851392 2509 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:11.851538 2509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:11.851859 2509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:11.854822 2509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "6cbe5353b3fb47398446cf676320e8cc"
format_stamp: "Formatted at 2026-09-24 16:04:11 on dist-test-slave-xn4h"
server_key: "de2735611dbdc5cabe5b9c64e866a6b9"
server_key_iv: "ad917b04e2b4dbd47f610ef4668029bb"
server_key_version: "encryptionkey@0"
I20260924 16:04:11.855674 2509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "6cbe5353b3fb47398446cf676320e8cc"
format_stamp: "Formatted at 2026-09-24 16:04:11 on dist-test-slave-xn4h"
server_key: "de2735611dbdc5cabe5b9c64e866a6b9"
server_key_iv: "ad917b04e2b4dbd47f610ef4668029bb"
server_key_version: "encryptionkey@0"
I20260924 16:04:11.861079 2509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:04:11.865104 2524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:11.866600 2509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:11.866791 2509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "6cbe5353b3fb47398446cf676320e8cc"
format_stamp: "Formatted at 2026-09-24 16:04:11 on dist-test-slave-xn4h"
server_key: "de2735611dbdc5cabe5b9c64e866a6b9"
server_key_iv: "ad917b04e2b4dbd47f610ef4668029bb"
server_key_version: "encryptionkey@0"
I20260924 16:04:11.867014 2509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:11.896662 2509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:11.902563 2509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:11.902877 2509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:11.903898 2509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:11.905345 2509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:11.905479 2509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:11.905638 2509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:11.905689 2509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:11.938282 2509 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:43579
I20260924 16:04:11.938393 2639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:43579 every 8 connection(s)
I20260924 16:04:11.939651 2509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:11.947414 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 2509
I20260924 16:04:11.947599 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:04:11.948077 17318 external_mini_cluster.cc:1428] Setting key f40d1f4b3797efe09471b64ec24c8c93
Sep 24 16:04:11 dist-test-slave-xn4h krb5kdc[2260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265851, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:11.960039 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:11.942262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:33939 (local address 127.16.233.190:33233)
0924 16:04:11.942465 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:11.942469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:11.943610 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 16:04:11.945438 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:11.945445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:11.945540 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:11.945650 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:11.947258 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:11.948101 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:11.949925 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:11.950426 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:11.950554 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:11.954210 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:11.954215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:11.954219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:11.954253 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:11.956910 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:11.957657 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:11.957664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:11.957668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:11.957802 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:11.958240 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:11.958246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:11.958249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:11.958583 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:11.958741 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:11.959412 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 16:04:11.959666 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:04:11.961102 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:11.942766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:33233 (local address 127.16.233.130:33939)
0924 16:04:11.943442 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:11.943492 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:11.945169 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:11.945882 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:11.945891 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:11.946458 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:11.947049 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:11.947074 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:11.948295 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:11.948299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:11.949765 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:11.949788 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:11.950660 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:11.950668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:11.950915 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:11.951624 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:11.951646 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:11.953941 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:11.957075 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:11.957084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:11.957100 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:11.957533 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:11.957919 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:11.957924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:11.957928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:11.958121 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:11.958711 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:11.958719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:11.959059 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 16:04:11.959324 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 16:04:11.959901 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":263,"threads_started":1}
I20260924 16:04:11.962633 2640 heartbeater.cc:347] Connected to a master server at 127.16.233.190:33233
I20260924 16:04:11.962958 2640 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:11.963819 2640 heartbeater.cc:511] Master 127.16.233.190:33233 requested a full tablet report, sending...
I20260924 16:04:11.965449 2307 ts_manager.cc:194] Registered new tserver with Master: 6cbe5353b3fb47398446cf676320e8cc (127.16.233.130:43579)
I20260924 16:04:11.966756 2307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:33939
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 16:04:12.004089 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:33233
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:12.338112 2645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:12.338551 2645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:12.338670 2645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:12.345892 2645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:12.345999 2645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:12.346108 2645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:04:12.356012 2645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:33233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:12.357254 2645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:12.358619 2645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:12.372391 2653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:12.372418 2651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:12.372493 2650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:12.373500 2645 server_base.cc:1034] running on GCE node
I20260924 16:04:12.374174 2645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:12.375281 2645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:12.376493 2645 hybrid_clock.cc:648] HybridClock initialized: now 1790265852376442 us; error 45 us; skew 500 ppm
I20260924 16:04:12.376822 2645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:12 dist-test-slave-xn4h krb5kdc[2260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265852, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:12.379980 2645 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:04:12.381451 2645 webserver.cc:460] Webserver started at http://127.16.233.131:38399/ using document root <none> and password file <none>
I20260924 16:04:12.382117 2645 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:12.382246 2645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:12.382587 2645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:12.385416 2645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "afbfc0939ad44ff1870ce23b71c954a1"
format_stamp: "Formatted at 2026-09-24 16:04:12 on dist-test-slave-xn4h"
server_key: "a403e6e1102664346c9bc521c7189361"
server_key_iv: "cf08041ab8ba8df9066cd6e26229b2aa"
server_key_version: "encryptionkey@0"
I20260924 16:04:12.386150 2645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "afbfc0939ad44ff1870ce23b71c954a1"
format_stamp: "Formatted at 2026-09-24 16:04:12 on dist-test-slave-xn4h"
server_key: "a403e6e1102664346c9bc521c7189361"
server_key_iv: "cf08041ab8ba8df9066cd6e26229b2aa"
server_key_version: "encryptionkey@0"
I20260924 16:04:12.391215 2645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:04:12.394707 2660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:12.396005 2645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:04:12.396191 2645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "afbfc0939ad44ff1870ce23b71c954a1"
format_stamp: "Formatted at 2026-09-24 16:04:12 on dist-test-slave-xn4h"
server_key: "a403e6e1102664346c9bc521c7189361"
server_key_iv: "cf08041ab8ba8df9066cd6e26229b2aa"
server_key_version: "encryptionkey@0"
I20260924 16:04:12.396418 2645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:12.420060 2645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:12.424418 2645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:12.424738 2645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:12.425697 2645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:12.426986 2645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:12.427091 2645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:12.427245 2645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:12.427316 2645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:12.453868 2503 heartbeater.cc:503] Master 127.16.233.190:33233 was elected leader, sending a full tablet report...
I20260924 16:04:12.460600 2645 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:36203
I20260924 16:04:12.460676 2775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:36203 every 8 connection(s)
I20260924 16:04:12.461936 2645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:12.469120 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 2645
I20260924 16:04:12.469326 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:04:12.469806 17318 external_mini_cluster.cc:1428] Setting key 8e29cccb3a0c4e1e46b1ef0bed32b94b
Sep 24 16:04:12 dist-test-slave-xn4h krb5kdc[2260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265852, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:12.481876 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.464580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:51965 (local address 127.16.233.190:33233)
0924 16:04:12.465079 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.465084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.466059 (+ 975us) server_negotiation.cc:407] Connection header received
0924 16:04:12.467980 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.467995 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.468084 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.468197 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:12.469726 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.470833 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.472900 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.473363 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.473457 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.476785 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:12.476790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:12.476793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:12.476824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:12.479280 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:12.479958 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:12.479961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:12.479964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:12.480075 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:12.480425 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:12.480428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:12.480430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:12.480682 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:12.480795 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.481444 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.481613 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260924 16:04:12.482897 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.465097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:33233 (local address 127.16.233.131:51965)
0924 16:04:12.465904 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:12.465948 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:12.467700 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:12.468428 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:12.468436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:12.468920 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:12.469559 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.469578 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.471002 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.471024 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.472740 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.472755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.473544 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.473549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.473746 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.474411 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:12.474434 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:12.476594 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:12.479397 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:12.479404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:12.479428 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:12.479824 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:12.480162 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:12.480165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:12.480168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:12.480337 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:12.480789 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:12.480795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:12.481118 (+ 323us) client_negotiation.cc:769] Sending connection context
0924 16:04:12.481356 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 16:04:12.481728 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":339,"threads_started":1}
I20260924 16:04:12.484402 2776 heartbeater.cc:347] Connected to a master server at 127.16.233.190:33233
I20260924 16:04:12.484766 2776 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:12.485661 2776 heartbeater.cc:511] Master 127.16.233.190:33233 requested a full tablet report, sending...
I20260924 16:04:12.487126 2306 ts_manager.cc:194] Registered new tserver with Master: afbfc0939ad44ff1870ce23b71c954a1 (127.16.233.131:36203)
I20260924 16:04:12.488121 2306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:51965
I20260924 16:04:12.497442 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:12.513913 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.502213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.16.233.190:33233)
0924 16:04:12.502426 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.502430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.502780 (+ 350us) server_negotiation.cc:407] Connection header received
0924 16:04:12.502911 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.502915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.502973 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.503058 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:12.504407 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.505097 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.506294 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.506748 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.506839 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.509156 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:12.509161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:12.509164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:12.509193 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:12.511768 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:12.512361 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:12.512365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:12.512367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:12.512474 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:12.512772 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:12.512775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:12.512777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:12.513073 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:12.513250 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.513535 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.513687 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:04:12.517740 2306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:12.524657 2306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:12.557410 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.536350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.16.233.130:43579)
0924 16:04:12.547153 (+ 10803us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.547166 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.548256 (+ 1090us) server_negotiation.cc:407] Connection header received
0924 16:04:12.548372 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.548383 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.548656 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.548851 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:12.549818 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.551501 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.554670 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.556156 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.556318 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.556488 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.556864 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.557063 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10566,"thread_start_us":10819,"threads_started":1}
I20260924 16:04:12.557453 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.536080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:43579 (local address 127.0.0.1:51986)
0924 16:04:12.547951 (+ 11871us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:12.547988 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:12.548170 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:12.549009 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:12.549019 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:12.549043 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:12.549512 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.549526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.551900 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.551907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.554504 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.554525 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.556400 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.556406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.556612 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.556669 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 16:04:12.556801 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 16:04:12.556983 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11639,"mutex_wait_us":63,"thread_start_us":12389,"threads_started":1}
I20260924 16:04:12.558166 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.536327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:36203 (local address 127.0.0.1:39964)
0924 16:04:12.548714 (+ 12387us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:12.548731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:12.548838 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:12.550973 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:12.550978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:12.551003 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:12.551325 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.551335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.553272 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.553277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.555423 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.555438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.557359 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.557365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.557593 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.557630 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:04:12.557710 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 16:04:12.557832 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12263,"mutex_wait_us":22,"thread_start_us":11235,"threads_started":1}
I20260924 16:04:12.558744 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.536956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:44515 (local address 127.0.0.1:42810)
0924 16:04:12.550804 (+ 13848us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:12.550818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:12.550952 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:12.551465 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:12.551469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:12.551494 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:12.551911 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.551924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.553636 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.553642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.555769 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.555782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.557915 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.557922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.558126 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.558160 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:04:12.558247 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:04:12.558385 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13726,"mutex_wait_us":75,"thread_start_us":13597,"threads_started":1}
I20260924 16:04:12.559424 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.536934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.16.233.131:36203)
0924 16:04:12.550113 (+ 13179us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.550120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.550164 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:04:12.550256 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.550263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.550517 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.550688 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:12.551534 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.553130 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.555615 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.556858 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.557041 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.557189 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.558906 (+ 1717us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.559074 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13008,"thread_start_us":12829,"threads_started":1}
I20260924 16:04:12.559600 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.537668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.16.233.129:44515)
0924 16:04:12.549582 (+ 11914us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.549590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.550915 (+ 1325us) server_negotiation.cc:407] Connection header received
0924 16:04:12.551121 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.551130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.551298 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.551470 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:12.552137 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.553486 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.555955 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.557392 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.557593 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.557761 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.559108 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.559277 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11738,"thread_start_us":11542,"threads_started":1}
I20260924 16:04:12.561708 2573 tablet_service.cc:1467] Processing CreateTablet for tablet f4b51bc1374149a99ca3f3951831c311 (DEFAULT_TABLE table=test-table [id=334748ccac564c549121cd5aa37c18a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:12.563279 2573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b51bc1374149a99ca3f3951831c311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:12.563053 2709 tablet_service.cc:1467] Processing CreateTablet for tablet f4b51bc1374149a99ca3f3951831c311 (DEFAULT_TABLE table=test-table [id=334748ccac564c549121cd5aa37c18a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:12.563390 2435 tablet_service.cc:1467] Processing CreateTablet for tablet f4b51bc1374149a99ca3f3951831c311 (DEFAULT_TABLE table=test-table [id=334748ccac564c549121cd5aa37c18a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:12.564394 2709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b51bc1374149a99ca3f3951831c311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:12.564713 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b51bc1374149a99ca3f3951831c311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:12.576407 2795 tablet_bootstrap.cc:492] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: Bootstrap starting.
I20260924 16:04:12.577888 2796 tablet_bootstrap.cc:492] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: Bootstrap starting.
I20260924 16:04:12.579676 2797 tablet_bootstrap.cc:492] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: Bootstrap starting.
I20260924 16:04:12.580102 2795 tablet_bootstrap.cc:654] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:12.581316 2796 tablet_bootstrap.cc:654] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:12.581447 2795 log.cc:824] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:12.582584 2796 log.cc:824] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:12.582751 2797 tablet_bootstrap.cc:654] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:12.583930 2797 log.cc:824] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:12.585601 2795 tablet_bootstrap.cc:492] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: No bootstrap required, opened a new log
I20260924 16:04:12.585939 2795 ts_tablet_manager.cc:1397] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 16:04:12.586423 2797 tablet_bootstrap.cc:492] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: No bootstrap required, opened a new log
I20260924 16:04:12.586684 2797 ts_tablet_manager.cc:1397] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:04:12.588857 2796 tablet_bootstrap.cc:492] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: No bootstrap required, opened a new log
I20260924 16:04:12.589129 2796 ts_tablet_manager.cc:1397] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260924 16:04:12.594051 2797 raft_consensus.cc:348] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } }
I20260924 16:04:12.594416 2797 raft_consensus.cc:374] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:12.594520 2797 raft_consensus.cc:729] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950c55210d1f4ec8bf99e04e3d0357b2, State: Initialized, Role: FOLLOWER
I20260924 16:04:12.595135 2797 consensus_queue.cc:260] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } }
I20260924 16:04:12.596516 2795 raft_consensus.cc:348] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } }
I20260924 16:04:12.596907 2795 raft_consensus.cc:374] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:12.597005 2795 raft_consensus.cc:729] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbfc0939ad44ff1870ce23b71c954a1, State: Initialized, Role: FOLLOWER
I20260924 16:04:12.597693 2795 consensus_queue.cc:260] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } }
I20260924 16:04:12.597913 2797 ts_tablet_manager.cc:1428] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260924 16:04:12.598991 2776 heartbeater.cc:503] Master 127.16.233.190:33233 was elected leader, sending a full tablet report...
I20260924 16:04:12.599584 2795 ts_tablet_manager.cc:1428] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260924 16:04:12.599715 2796 raft_consensus.cc:348] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } }
I20260924 16:04:12.600553 2796 raft_consensus.cc:374] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:12.600658 2796 raft_consensus.cc:729] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cbe5353b3fb47398446cf676320e8cc, State: Initialized, Role: FOLLOWER
I20260924 16:04:12.601222 2796 consensus_queue.cc:260] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } }
I20260924 16:04:12.602385 2640 heartbeater.cc:503] Master 127.16.233.190:33233 was elected leader, sending a full tablet report...
I20260924 16:04:12.603039 2796 ts_tablet_manager.cc:1428] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260924 16:04:12.637276 2802 raft_consensus.cc:482] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:12.637674 2802 raft_consensus.cc:504] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } }
I20260924 16:04:12.639635 2802 leader_election.cc:288] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cbe5353b3fb47398446cf676320e8cc (127.16.233.130:43579), 950c55210d1f4ec8bf99e04e3d0357b2 (127.16.233.129:44515)
I20260924 16:04:12.646143 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.639870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:43579 (local address 127.16.233.131:47887)
0924 16:04:12.640110 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:12.640131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:12.640281 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:12.640773 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:12.640781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:12.640819 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:12.641009 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.641047 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.642413 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.642419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.643907 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.643921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.645328 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.645334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.645536 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.645573 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:04:12.645669 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 16:04:12.645811 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 16:04:12.646129 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.640213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:47887 (local address 127.16.233.130:43579)
0924 16:04:12.640436 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.640441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.640464 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:04:12.640543 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.640550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.640640 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.640741 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:12.641212 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.642257 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.644078 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.645172 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.645292 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.645345 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.645711 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.645810 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:04:12.646778 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.640434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:50393 (local address 127.16.233.129:44515)
0924 16:04:12.640667 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.640674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.641075 (+ 401us) server_negotiation.cc:407] Connection header received
0924 16:04:12.641163 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.641170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.641254 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.641337 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:12.641950 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.642866 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.644777 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.645802 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.645951 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.646028 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.646441 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.646597 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:04:12.647312 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.640187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:44515 (local address 127.16.233.131:50393)
0924 16:04:12.640959 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:12.640974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:12.641072 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:12.641381 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:12.641385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:12.641410 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:12.641791 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.641803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.643018 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.643022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.644618 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:12.644632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.646011 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.646018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:12.646205 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.646238 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:04:12.646975 (+ 737us) client_negotiation.cc:240] Negotiation successful
0924 16:04:12.647081 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":416,"threads_started":1}
I20260924 16:04:12.647305 2594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b51bc1374149a99ca3f3951831c311" candidate_uuid: "afbfc0939ad44ff1870ce23b71c954a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbe5353b3fb47398446cf676320e8cc" is_pre_election: true
I20260924 16:04:12.647891 2594 raft_consensus.cc:2454] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afbfc0939ad44ff1870ce23b71c954a1 in term 0.
I20260924 16:04:12.648514 2457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b51bc1374149a99ca3f3951831c311" candidate_uuid: "afbfc0939ad44ff1870ce23b71c954a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" is_pre_election: true
I20260924 16:04:12.648720 2662 leader_election.cc:302] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cbe5353b3fb47398446cf676320e8cc, afbfc0939ad44ff1870ce23b71c954a1; no voters:
I20260924 16:04:12.649147 2802 raft_consensus.cc:2781] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:12.649130 2457 raft_consensus.cc:2454] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afbfc0939ad44ff1870ce23b71c954a1 in term 0.
I20260924 16:04:12.649331 2802 raft_consensus.cc:482] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:12.649439 2802 raft_consensus.cc:3037] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:12.652079 2802 raft_consensus.cc:504] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } }
I20260924 16:04:12.652896 2802 leader_election.cc:288] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [CANDIDATE]: Term 1 election: Requested vote from peers 6cbe5353b3fb47398446cf676320e8cc (127.16.233.130:43579), 950c55210d1f4ec8bf99e04e3d0357b2 (127.16.233.129:44515)
I20260924 16:04:12.653360 2594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b51bc1374149a99ca3f3951831c311" candidate_uuid: "afbfc0939ad44ff1870ce23b71c954a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbe5353b3fb47398446cf676320e8cc"
I20260924 16:04:12.653412 2457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b51bc1374149a99ca3f3951831c311" candidate_uuid: "afbfc0939ad44ff1870ce23b71c954a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950c55210d1f4ec8bf99e04e3d0357b2"
I20260924 16:04:12.653616 2594 raft_consensus.cc:3037] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:12.653625 2457 raft_consensus.cc:3037] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:12.657024 2457 raft_consensus.cc:2454] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbfc0939ad44ff1870ce23b71c954a1 in term 1.
I20260924 16:04:12.657025 2594 raft_consensus.cc:2454] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbfc0939ad44ff1870ce23b71c954a1 in term 1.
I20260924 16:04:12.657545 2664 leader_election.cc:302] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 950c55210d1f4ec8bf99e04e3d0357b2, afbfc0939ad44ff1870ce23b71c954a1; no voters:
I20260924 16:04:12.657956 2802 raft_consensus.cc:2781] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:12.658483 2802 raft_consensus.cc:686] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 1 LEADER]: Becoming Leader. State: Replica: afbfc0939ad44ff1870ce23b71c954a1, State: Running, Role: LEADER
I20260924 16:04:12.658980 2802 consensus_queue.cc:237] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } }
I20260924 16:04:12.663803 2305 catalog_manager.cc:5495] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to afbfc0939ad44ff1870ce23b71c954a1 (127.16.233.131). New cstate: current_term: 1 leader_uuid: "afbfc0939ad44ff1870ce23b71c954a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afbfc0939ad44ff1870ce23b71c954a1" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 36203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 } health_report { overall_health: UNKNOWN } } }
W20260924 16:04:12.673267 2504 tablet.cc:2369] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:04:12.688946 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.683456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.16.233.131:36203)
0924 16:04:12.683674 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.683681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.683700 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:04:12.683889 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.683893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.683980 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.684139 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:12.684520 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.685260 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.686774 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.687188 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.687328 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.687836 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:12.688353 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:12.688534 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.688590 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.688675 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 16:04:12.691851 2641 tablet.cc:2369] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:04:12.698132 2594 raft_consensus.cc:1261] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [term 1 FOLLOWER]: Refusing update from remote peer afbfc0939ad44ff1870ce23b71c954a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:12.698133 2457 raft_consensus.cc:1261] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [term 1 FOLLOWER]: Refusing update from remote peer afbfc0939ad44ff1870ce23b71c954a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:12.699323 2805 consensus_queue.cc:1035] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "950c55210d1f4ec8bf99e04e3d0357b2" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:12.699810 2802 consensus_queue.cc:1035] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cbe5353b3fb47398446cf676320e8cc" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:12.711524 2808 mvcc.cc:205] Tried to move back new op lower bound from 7332928932639969280 to 7332928932495396864. Current Snapshot: MvccSnapshot[applied={T|T < 7332928932639969280}]
I20260924 16:04:12.716666 2809 mvcc.cc:205] Tried to move back new op lower bound from 7332928932639969280 to 7332928932495396864. Current Snapshot: MvccSnapshot[applied={T|T < 7332928932639969280}]
I20260924 16:04:12.724902 2305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:12.725205 2305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:12.729023 2305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e517c1875ddf483baf63f84b93392c68: Sending DeleteTablet for 3 replicas of tablet f4b51bc1374149a99ca3f3951831c311
I20260924 16:04:12.730294 2709 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b51bc1374149a99ca3f3951831c311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:12 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:39964
I20260924 16:04:12.730293 2573 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b51bc1374149a99ca3f3951831c311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:12 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:51986
I20260924 16:04:12.730437 2435 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b51bc1374149a99ca3f3951831c311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:12 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:42810
I20260924 16:04:12.732908 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 2370
I20260924 16:04:12.753891 2819 tablet_replica.cc:330] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: stopping tablet replica
I20260924 16:04:12.754712 2819 raft_consensus.cc:2229] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:12.755530 2819 raft_consensus.cc:2258] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 16:04:12 dist-test-slave-xn4h krb5kdc[2260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 16:04:12.765657 2818 tablet_replica.cc:330] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: stopping tablet replica
I20260924 16:04:12.765776 2819 ts_tablet_manager.cc:1905] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:12.765313 2817 tablet_replica.cc:330] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: stopping tablet replica
I20260924 16:04:12.766738 2817 raft_consensus.cc:2229] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:04:12.767019 2818 raft_consensus.cc:2229] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:12.767784 2817 raft_consensus.cc:2258] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:12.768371 2818 raft_consensus.cc:2258] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:12.772046 2818 ts_tablet_manager.cc:1905] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:12.772683 2817 ts_tablet_manager.cc:1905] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:12.775151 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.734821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.16.233.129:44515)
0924 16:04:12.735071 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.735080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.745659 (+ 10579us) server_negotiation.cc:407] Connection header received
0924 16:04:12.746293 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.746341 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.746462 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.746564 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:12.749090 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.750696 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.755818 (+ 5122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.756802 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.756953 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.766450 (+ 9497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:12.766468 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:12.766486 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:12.766538 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:12.770828 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:12.772073 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:12.772091 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:12.772096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:12.772301 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:12.772841 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:12.772848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:12.772853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:12.773380 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:12.773718 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.774434 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.774714 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:04:12.775199 2819 ts_tablet_manager.cc:1918] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:12.775381 2819 log.cc:1196] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/f4b51bc1374149a99ca3f3951831c311
I20260924 16:04:12.775959 2819 ts_tablet_manager.cc:1939] T f4b51bc1374149a99ca3f3951831c311 P 950c55210d1f4ec8bf99e04e3d0357b2: Deleting consensus metadata
I20260924 16:04:12.776759 2498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:12.783586 2818 ts_tablet_manager.cc:1918] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:12.784365 2818 log.cc:1196] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/f4b51bc1374149a99ca3f3951831c311
I20260924 16:04:12.785053 2818 ts_tablet_manager.cc:1939] T f4b51bc1374149a99ca3f3951831c311 P 6cbe5353b3fb47398446cf676320e8cc: Deleting consensus metadata
I20260924 16:04:12.787250 2290 catalog_manager.cc:4841] TS 6cbe5353b3fb47398446cf676320e8cc (127.16.233.130:43579): tablet f4b51bc1374149a99ca3f3951831c311 (table test-table [id=334748ccac564c549121cd5aa37c18a4]) successfully deleted
I20260924 16:04:12.787426 2817 ts_tablet_manager.cc:1918] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:12.787611 2817 log.cc:1196] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/f4b51bc1374149a99ca3f3951831c311
I20260924 16:04:12.788163 2817 ts_tablet_manager.cc:1939] T f4b51bc1374149a99ca3f3951831c311 P afbfc0939ad44ff1870ce23b71c954a1: Deleting consensus metadata
I20260924 16:04:12.789979 2291 catalog_manager.cc:4841] TS afbfc0939ad44ff1870ce23b71c954a1 (127.16.233.131:36203): tablet f4b51bc1374149a99ca3f3951831c311 (table test-table [id=334748ccac564c549121cd5aa37c18a4]) successfully deleted
I20260924 16:04:12.889953 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 2370
I20260924 16:04:12.889968 2293 catalog_manager.cc:4841] TS 950c55210d1f4ec8bf99e04e3d0357b2 (127.16.233.129:44515): tablet f4b51bc1374149a99ca3f3951831c311 (table test-table [id=334748ccac564c549121cd5aa37c18a4]) successfully deleted
I20260924 16:04:12.910698 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 2509
Sep 24 16:04:12 dist-test-slave-xn4h krb5kdc[2260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:04:12.927685 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:12.911574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.16.233.130:43579)
0924 16:04:12.911771 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:12.911778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:12.911802 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:04:12.911895 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:12.911901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:12.912003 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:12.912116 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:12.913322 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.914490 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.915899 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:12.916548 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:12.916678 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:12.920966 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:12.920993 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:12.921019 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:12.921070 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:12.924786 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:12.925482 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:12.925491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:12.925495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:12.925643 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:12.925979 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:12.925991 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:12.925994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:12.926511 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:12.926802 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:12.927090 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 16:04:12.927315 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:04:12.928411 2635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:13.018184 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 2509
I20260924 16:04:13.035720 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 2645
Sep 24 16:04:13 dist-test-slave-xn4h krb5kdc[2260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 16:04:13.053692 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:13.036528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.16.233.131:36203)
0924 16:04:13.036731 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:13.036738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:13.036760 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:04:13.036904 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:13.036910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:13.037031 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:13.037157 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:13.038569 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:13.039780 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:13.041050 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:13.041754 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:13.041883 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:13.046623 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:13.046642 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:13.046663 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:13.046714 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:13.050792 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:13.051421 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:13.051436 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:13.051440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:13.051656 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:13.052024 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:13.052029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:13.052033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:13.052515 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:13.052819 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:13.053038 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 16:04:13.053280 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:04:13.054476 2771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:13.179277 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 2645
I20260924 16:04:13.199257 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 2273
I20260924 16:04:13.200421 2339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:13.310859 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 2273
2026-09-24T16:04:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4347 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T16:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:13Z Disabled control of system clock
I20260924 16:04:13.389704 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41259
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:41649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:41259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:04:13.718003 2828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:13.718297 2828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:13.718513 2828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:13.725924 2828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:13.726038 2828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:13.726123 2828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:13.726233 2828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:13.726332 2828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:13.735793 2828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:41259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:13.737083 2828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:13.738900 2828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:13.750875 2835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:13.750900 2834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:13.750840 2837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:13.751750 2828 server_base.cc:1034] running on GCE node
I20260924 16:04:13.752526 2828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:13.753947 2828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:13.755172 2828 hybrid_clock.cc:648] HybridClock initialized: now 1790265853755133 us; error 50 us; skew 500 ppm
I20260924 16:04:13.755540 2828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:13.757838 2828 webserver.cc:460] Webserver started at http://127.16.233.190:39375/ using document root <none> and password file <none>
I20260924 16:04:13.758622 2828 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:13.758756 2828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:13.759084 2828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:13.762096 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "c061932a7cf94241a78bbc3544fee8c9"
format_stamp: "Formatted at 2026-09-24 16:04:13 on dist-test-slave-xn4h"
server_key: "86da1a0fe7a89efcae7d1e0e3ac8a792"
server_key_iv: "1ba3395f346fff51dd0f8d413d0f857b"
server_key_version: "encryptionkey@0"
I20260924 16:04:13.762977 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "c061932a7cf94241a78bbc3544fee8c9"
format_stamp: "Formatted at 2026-09-24 16:04:13 on dist-test-slave-xn4h"
server_key: "86da1a0fe7a89efcae7d1e0e3ac8a792"
server_key_iv: "1ba3395f346fff51dd0f8d413d0f857b"
server_key_version: "encryptionkey@0"
I20260924 16:04:13.767956 2828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:04:13.771337 2843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:13.772662 2828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:13.772846 2828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "c061932a7cf94241a78bbc3544fee8c9"
format_stamp: "Formatted at 2026-09-24 16:04:13 on dist-test-slave-xn4h"
server_key: "86da1a0fe7a89efcae7d1e0e3ac8a792"
server_key_iv: "1ba3395f346fff51dd0f8d413d0f857b"
server_key_version: "encryptionkey@0"
I20260924 16:04:13.773075 2828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:13.785687 2828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:13.786617 2828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:13.786902 2828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:13.805457 2828 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:41259
I20260924 16:04:13.805477 2894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:41259 every 8 connection(s)
I20260924 16:04:13.806880 2828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:13.810822 2896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:13.811997 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 2828
I20260924 16:04:13.812198 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:13.812831 17318 external_mini_cluster.cc:1428] Setting key acf03025cd82b4d68457342410e28db8
I20260924 16:04:13.823870 2896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9: Bootstrap starting.
I20260924 16:04:13.827052 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:13.814853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.16.233.190:41259)
0924 16:04:13.815861 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:13.815872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:13.815916 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:04:13.817870 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:13.817893 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:13.818425 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:13.818835 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:13.819375 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:13.820604 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:13.822057 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:13.822789 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:13.822971 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:13.824196 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:13.824206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:13.824222 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:13.824905 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:13.825017 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:13.825043 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:13.825177 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:13.825390 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 16:04:13.825764 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":336,"threads_started":1}
I20260924 16:04:13.827910 2896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:13.828989 2896 log.cc:824] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:13.831737 2896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9: No bootstrap required, opened a new log
I20260924 16:04:13.838644 2896 raft_consensus.cc:348] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c061932a7cf94241a78bbc3544fee8c9" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41259 } }
I20260924 16:04:13.838989 2896 raft_consensus.cc:374] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:13.839084 2896 raft_consensus.cc:729] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c061932a7cf94241a78bbc3544fee8c9, State: Initialized, Role: FOLLOWER
I20260924 16:04:13.839751 2896 consensus_queue.cc:260] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c061932a7cf94241a78bbc3544fee8c9" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41259 } }
I20260924 16:04:13.840083 2896 raft_consensus.cc:388] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:13.840233 2896 raft_consensus.cc:482] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:13.840395 2896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:13.842607 2896 raft_consensus.cc:504] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c061932a7cf94241a78bbc3544fee8c9" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41259 } }
I20260924 16:04:13.843130 2896 leader_election.cc:302] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c061932a7cf94241a78bbc3544fee8c9; no voters:
I20260924 16:04:13.843706 2896 leader_election.cc:288] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:13.843911 2901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:13.844607 2901 raft_consensus.cc:686] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [term 1 LEADER]: Becoming Leader. State: Replica: c061932a7cf94241a78bbc3544fee8c9, State: Running, Role: LEADER
I20260924 16:04:13.845098 2901 consensus_queue.cc:237] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c061932a7cf94241a78bbc3544fee8c9" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41259 } }
I20260924 16:04:13.845844 2896 sys_catalog.cc:564] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:13.849022 2903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c061932a7cf94241a78bbc3544fee8c9. Latest consensus state: current_term: 1 leader_uuid: "c061932a7cf94241a78bbc3544fee8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c061932a7cf94241a78bbc3544fee8c9" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41259 } } }
I20260924 16:04:13.849227 2902 sys_catalog.cc:455] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c061932a7cf94241a78bbc3544fee8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c061932a7cf94241a78bbc3544fee8c9" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41259 } } }
I20260924 16:04:13.849560 2902 sys_catalog.cc:458] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:13.849565 2903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:13.854959 2908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:13.860386 2908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:13.869709 2908 catalog_manager.cc:1347] Generated new cluster ID: e6e09136c5e14d4fa61ac837be7bd873
I20260924 16:04:13.869853 2908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:13.883471 2908 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:13.884801 2908 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:13.890473 2908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9: Generated new TSK 0
I20260924 16:04:13.891551 2908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:13.899518 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41259
--builtin_ntp_servers=127.16.233.148:41649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:04:14.245564 2920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:14.245899 2920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:14.246006 2920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:14.253456 2920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:14.253612 2920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:14.253727 2920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:04:14.263870 2920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:14.265219 2920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:14.267019 2920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:14.280471 2926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:14.280414 2925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:14.280511 2928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:14.281232 2920 server_base.cc:1034] running on GCE node
I20260924 16:04:14.281884 2920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:14.282868 2920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:14.284080 2920 hybrid_clock.cc:648] HybridClock initialized: now 1790265854284011 us; error 32 us; skew 500 ppm
I20260924 16:04:14.284479 2920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:14.287529 2920 webserver.cc:460] Webserver started at http://127.16.233.129:33555/ using document root <none> and password file <none>
I20260924 16:04:14.288300 2920 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:14.288427 2920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:14.288702 2920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:14.291543 2920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "32d85603163849a6ad2fac105650b358"
format_stamp: "Formatted at 2026-09-24 16:04:14 on dist-test-slave-xn4h"
server_key: "ad93a7cb4dc9f483d97d2f3bda89a022"
server_key_iv: "59a0685a923126fda41a3c65d96432eb"
server_key_version: "encryptionkey@0"
I20260924 16:04:14.292294 2920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "32d85603163849a6ad2fac105650b358"
format_stamp: "Formatted at 2026-09-24 16:04:14 on dist-test-slave-xn4h"
server_key: "ad93a7cb4dc9f483d97d2f3bda89a022"
server_key_iv: "59a0685a923126fda41a3c65d96432eb"
server_key_version: "encryptionkey@0"
I20260924 16:04:14.297387 2920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:04:14.300885 2934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:14.302129 2920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:04:14.302372 2920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "32d85603163849a6ad2fac105650b358"
format_stamp: "Formatted at 2026-09-24 16:04:14 on dist-test-slave-xn4h"
server_key: "ad93a7cb4dc9f483d97d2f3bda89a022"
server_key_iv: "59a0685a923126fda41a3c65d96432eb"
server_key_version: "encryptionkey@0"
I20260924 16:04:14.302603 2920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:14.316749 2920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:14.317582 2920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:14.317883 2920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:14.319022 2920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:14.320472 2920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:14.320564 2920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:14.320644 2920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:14.320698 2920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:14.354688 2920 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:36089
I20260924 16:04:14.354754 3047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:36089 every 8 connection(s)
I20260924 16:04:14.356211 2920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:14.356598 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 2920
I20260924 16:04:14.356812 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:04:14.357393 17318 external_mini_cluster.cc:1428] Setting key 87b98de167e3dea9f3570511f0a38a08
I20260924 16:04:14.360574 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41259
--builtin_ntp_servers=127.16.233.148:41649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:04:14.378111 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:14.359125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:57003 (local address 127.16.233.190:41259)
0924 16:04:14.359617 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:14.359623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:14.367434 (+ 7811us) server_negotiation.cc:407] Connection header received
0924 16:04:14.370118 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:14.370123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:14.370194 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:14.370295 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:14.371760 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:14.372443 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:14.375033 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:14.375428 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:14.375529 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:14.377318 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:14.377324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:14.377327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:14.377346 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:14.377385 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:14.377397 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:14.377464 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:14.377767 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 16:04:14.377860 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":351,"threads_started":1}
I20260924 16:04:14.379621 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:14.359848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41259 (local address 127.16.233.129:57003)
0924 16:04:14.367263 (+ 7415us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:14.367309 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:14.369811 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:14.370592 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:14.370606 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:14.370738 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:14.371578 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:14.371608 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:14.372613 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:14.372621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:14.374890 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:14.374909 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:14.375666 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:14.375676 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:14.376038 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:14.376890 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:04:14.376896 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:14.376951 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:04:14.377057 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:14.377495 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:14.377509 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:14.377520 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:04:14.377721 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 16:04:14.378157 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6796,"thread_start_us":6573,"threads_started":1}
I20260924 16:04:14.381333 3048 heartbeater.cc:347] Connected to a master server at 127.16.233.190:41259
I20260924 16:04:14.381666 3048 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:14.382472 3048 heartbeater.cc:511] Master 127.16.233.190:41259 requested a full tablet report, sending...
I20260924 16:04:14.384924 2860 ts_manager.cc:194] Registered new tserver with Master: 32d85603163849a6ad2fac105650b358 (127.16.233.129:36089)
I20260924 16:04:14.387384 2860 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.129:57003
W20260924 16:04:14.697494 3052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:14.697916 3052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:14.698060 3052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:14.706269 3052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:14.706421 3052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:14.706627 3052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:04:14.715833 3052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:14.717087 3052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:14.718909 3052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:14.731791 3057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:14.731828 3058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:14.731763 3060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:14.732865 3052 server_base.cc:1034] running on GCE node
I20260924 16:04:14.733379 3052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:14.734213 3052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:14.735482 3052 hybrid_clock.cc:648] HybridClock initialized: now 1790265854735452 us; error 66 us; skew 500 ppm
I20260924 16:04:14.735836 3052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:14.738023 3052 webserver.cc:460] Webserver started at http://127.16.233.130:38897/ using document root <none> and password file <none>
I20260924 16:04:14.738780 3052 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:14.738915 3052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:14.739239 3052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:14.742115 3052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "a93ca95159084b989a41c826391c25d3"
format_stamp: "Formatted at 2026-09-24 16:04:14 on dist-test-slave-xn4h"
server_key: "0083eaa79755bdf5f9d8910026ed1fa9"
server_key_iv: "27ff24243d667df143ce0ef961353f9a"
server_key_version: "encryptionkey@0"
I20260924 16:04:14.742978 3052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "a93ca95159084b989a41c826391c25d3"
format_stamp: "Formatted at 2026-09-24 16:04:14 on dist-test-slave-xn4h"
server_key: "0083eaa79755bdf5f9d8910026ed1fa9"
server_key_iv: "27ff24243d667df143ce0ef961353f9a"
server_key_version: "encryptionkey@0"
I20260924 16:04:14.748500 3052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:04:14.751991 3067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:14.753309 3052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:04:14.753485 3052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "a93ca95159084b989a41c826391c25d3"
format_stamp: "Formatted at 2026-09-24 16:04:14 on dist-test-slave-xn4h"
server_key: "0083eaa79755bdf5f9d8910026ed1fa9"
server_key_iv: "27ff24243d667df143ce0ef961353f9a"
server_key_version: "encryptionkey@0"
I20260924 16:04:14.753719 3052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:14.770202 3052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:14.771112 3052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:14.771440 3052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:14.772396 3052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:14.773836 3052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:14.773949 3052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:14.774061 3052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:14.774210 3052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:14.807871 3052 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:36115
I20260924 16:04:14.807914 3184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:36115 every 8 connection(s)
I20260924 16:04:14.809152 3052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:14.818502 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 3052
I20260924 16:04:14.818675 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:04:14.819217 17318 external_mini_cluster.cc:1428] Setting key 2aa9c08dbd7f97dfd3f2bb2a0cc73583
I20260924 16:04:14.822376 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41259
--builtin_ntp_servers=127.16.233.148:41649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:04:14.823261 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:14.811803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:44045 (local address 127.16.233.190:41259)
0924 16:04:14.812010 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:14.812015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:14.813508 (+ 1493us) server_negotiation.cc:407] Connection header received
0924 16:04:14.815928 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:14.815932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:14.815998 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:14.816087 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:14.817362 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:14.818117 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:14.819960 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:14.820647 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:14.820769 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:14.822341 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:14.822348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:14.822353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:14.822373 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:14.822414 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:14.822427 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:14.822531 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:14.822822 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 16:04:14.822925 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:04:14.824340 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:14.812340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41259 (local address 127.16.233.130:44045)
0924 16:04:14.813332 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:14.813374 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:14.815641 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:14.816325 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:14.816336 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:14.816457 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:14.817192 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:14.817215 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:14.818275 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:14.818282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:14.819761 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:14.819797 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:14.820883 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:14.820890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:14.821229 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:14.821995 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:04:14.821998 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:14.822035 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:04:14.822119 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:14.822605 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:14.822616 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:14.822623 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:04:14.822806 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 16:04:14.823214 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":283,"threads_started":1}
I20260924 16:04:14.825744 3185 heartbeater.cc:347] Connected to a master server at 127.16.233.190:41259
I20260924 16:04:14.826058 3185 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:14.826879 3185 heartbeater.cc:511] Master 127.16.233.190:41259 requested a full tablet report, sending...
I20260924 16:04:14.828266 2860 ts_manager.cc:194] Registered new tserver with Master: a93ca95159084b989a41c826391c25d3 (127.16.233.130:36115)
I20260924 16:04:14.829079 2860 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.130:44045
W20260924 16:04:15.155943 3188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:15.156226 3188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:15.156363 3188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:15.166065 3188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:15.166173 3188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:15.166415 3188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:04:15.175679 3188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:15.177145 3188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:15.179060 3188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:15.192327 3196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:15.192409 3193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:15.192376 3194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:15.193133 3188 server_base.cc:1034] running on GCE node
I20260924 16:04:15.193706 3188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:15.194687 3188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:15.195917 3188 hybrid_clock.cc:648] HybridClock initialized: now 1790265855195871 us; error 50 us; skew 500 ppm
I20260924 16:04:15.196278 3188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:15.198597 3188 webserver.cc:460] Webserver started at http://127.16.233.131:35207/ using document root <none> and password file <none>
I20260924 16:04:15.199280 3188 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:15.199414 3188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:15.199733 3188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:15.202636 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "604141e35cca4e03a7f5b59113b6398f"
format_stamp: "Formatted at 2026-09-24 16:04:15 on dist-test-slave-xn4h"
server_key: "5db6385c4f4d97fdc9f64b0b9d02b685"
server_key_iv: "200710914479bdf5c62473c5ee0b049a"
server_key_version: "encryptionkey@0"
I20260924 16:04:15.203420 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "604141e35cca4e03a7f5b59113b6398f"
format_stamp: "Formatted at 2026-09-24 16:04:15 on dist-test-slave-xn4h"
server_key: "5db6385c4f4d97fdc9f64b0b9d02b685"
server_key_iv: "200710914479bdf5c62473c5ee0b049a"
server_key_version: "encryptionkey@0"
I20260924 16:04:15.208528 3188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 16:04:15.211899 3202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:15.213189 3188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:04:15.213364 3188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "604141e35cca4e03a7f5b59113b6398f"
format_stamp: "Formatted at 2026-09-24 16:04:15 on dist-test-slave-xn4h"
server_key: "5db6385c4f4d97fdc9f64b0b9d02b685"
server_key_iv: "200710914479bdf5c62473c5ee0b049a"
server_key_version: "encryptionkey@0"
I20260924 16:04:15.213580 3188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:15.227329 3188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:15.228130 3188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:15.228434 3188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:15.229418 3188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:15.230819 3188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:15.230947 3188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:15.231076 3188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:15.231161 3188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:15.264127 3188 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:35343
I20260924 16:04:15.264175 3317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:35343 every 8 connection(s)
I20260924 16:04:15.265462 3188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:15.267676 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 3188
I20260924 16:04:15.267851 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:04:15.268297 17318 external_mini_cluster.cc:1428] Setting key 779c12766567bdd7e3dc6121b7289caf
I20260924 16:04:15.278023 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.269263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:43647 (local address 127.16.233.190:41259)
0924 16:04:15.269455 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.269462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.270126 (+ 664us) server_negotiation.cc:407] Connection header received
0924 16:04:15.272083 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.272087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.272152 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.272235 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:15.273191 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.273930 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.275409 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.275799 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.275947 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.277246 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:15.277251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:15.277254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:15.277270 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:15.277299 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:15.277310 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:15.277384 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.277703 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.277799 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 16:04:15.278847 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.268623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41259 (local address 127.16.233.131:43647)
0924 16:04:15.269942 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.269996 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:15.271833 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:15.272427 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:15.272435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:15.272514 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:15.273056 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.273072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.274076 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.274079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.275289 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.275301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.276000 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.276005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.276257 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.276931 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:04:15.276933 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:15.276969 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:04:15.277043 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:15.277399 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:15.277407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:15.277413 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:04:15.277551 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 16:04:15.277811 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":348,"threads_started":1}
I20260924 16:04:15.280079 3319 heartbeater.cc:347] Connected to a master server at 127.16.233.190:41259
I20260924 16:04:15.280347 3319 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:15.281126 3319 heartbeater.cc:511] Master 127.16.233.190:41259 requested a full tablet report, sending...
I20260924 16:04:15.282652 2860 ts_manager.cc:194] Registered new tserver with Master: 604141e35cca4e03a7f5b59113b6398f (127.16.233.131:35343)
I20260924 16:04:15.283505 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:15.283489 2860 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.131:43647
I20260924 16:04:15.293531 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.288041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.16.233.190:41259)
0924 16:04:15.288209 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.288213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.288699 (+ 486us) server_negotiation.cc:407] Connection header received
0924 16:04:15.288860 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.288864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.288926 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.289008 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:15.289467 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.290108 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.291424 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.291791 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.291876 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.292994 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:15.292998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:15.293001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:15.293017 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:15.293064 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:15.293075 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:15.293142 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.293266 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.293334 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:04:15.296922 2860 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:15.303577 2860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:15.331012 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.315847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.16.233.130:36115)
0924 16:04:15.316729 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.316735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.316761 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:04:15.316964 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.316973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.317250 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.317452 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:15.319794 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.321829 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.328264 (+ 6435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.329807 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.329977 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.330201 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.330510 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.330675 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":586,"threads_started":1}
I20260924 16:04:15.331048 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.315455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:36115 (local address 127.0.0.1:42876)
0924 16:04:15.316527 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.316574 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:15.316730 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:15.317780 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:15.317790 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:15.317826 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:15.319549 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.319570 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.323244 (+ 3674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.323253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.328024 (+ 4771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.328044 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.330007 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.330013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.330204 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.330261 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 16:04:15.330461 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 16:04:15.330660 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"spinlock_wait_cycles":3328,"thread_start_us":708,"threads_started":1}
I20260924 16:04:15.334996 3116 tablet_service.cc:1467] Processing CreateTablet for tablet 0891aede24294491a50acc9beb551313 (DEFAULT_TABLE table=test-table [id=3db4e289ff9849528b820b674e931001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:15.336292 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.316962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:37918)
0924 16:04:15.317684 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.317703 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:15.317860 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:15.329160 (+ 11300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:15.329164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:15.329188 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:15.329594 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.329604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.331920 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.331924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.333726 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.333739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.335443 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.335451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.335643 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.335679 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:04:15.335776 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:04:15.335910 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":44,"thread_start_us":298,"threads_started":1}
I20260924 16:04:15.336387 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.317257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.16.233.129:36089)
0924 16:04:15.328668 (+ 11411us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.328674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.328697 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:04:15.328785 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.328791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.328961 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.329163 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:15.329775 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.331763 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.333891 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.335190 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.335412 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.335657 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.335841 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.336010 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11255,"thread_start_us":11144,"threads_started":1}
I20260924 16:04:15.336577 3116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0891aede24294491a50acc9beb551313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:15.337495 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.317641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:35343 (local address 127.0.0.1:44814)
0924 16:04:15.328303 (+ 10662us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.328320 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:15.328439 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:15.329172 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:15.329178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:15.329205 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:15.329618 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.329634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.331755 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.331760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.334083 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.334099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.336308 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.336315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.336905 (+ 590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.336936 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:04:15.337043 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:04:15.337161 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10540,"mutex_wait_us":34,"thread_start_us":10381,"threads_started":1}
I20260924 16:04:15.337615 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.318270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.16.233.131:35343)
0924 16:04:15.328071 (+ 9801us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.328080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.328561 (+ 481us) server_negotiation.cc:407] Connection header received
0924 16:04:15.328675 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.328683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.328952 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.329851 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:15.330121 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.331625 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.334281 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.335765 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.335965 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.336171 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.337086 (+ 915us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.337233 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9576,"thread_start_us":9368,"threads_started":1}
I20260924 16:04:15.339895 2982 tablet_service.cc:1467] Processing CreateTablet for tablet 0891aede24294491a50acc9beb551313 (DEFAULT_TABLE table=test-table [id=3db4e289ff9849528b820b674e931001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:15.340425 3252 tablet_service.cc:1467] Processing CreateTablet for tablet 0891aede24294491a50acc9beb551313 (DEFAULT_TABLE table=test-table [id=3db4e289ff9849528b820b674e931001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:15.341082 2982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0891aede24294491a50acc9beb551313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:15.341449 3252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0891aede24294491a50acc9beb551313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:15.352444 3337 tablet_bootstrap.cc:492] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: Bootstrap starting.
I20260924 16:04:15.357558 3337 tablet_bootstrap.cc:654] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:15.358788 3337 log.cc:824] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:15.360599 3338 tablet_bootstrap.cc:492] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358: Bootstrap starting.
I20260924 16:04:15.367192 3338 tablet_bootstrap.cc:654] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:15.368965 3338 log.cc:824] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:15.369853 3340 tablet_bootstrap.cc:492] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: Bootstrap starting.
I20260924 16:04:15.373641 3340 tablet_bootstrap.cc:654] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:15.374289 3338 tablet_bootstrap.cc:492] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358: No bootstrap required, opened a new log
I20260924 16:04:15.374598 3337 tablet_bootstrap.cc:492] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: No bootstrap required, opened a new log
I20260924 16:04:15.374617 3338 ts_tablet_manager.cc:1397] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20260924 16:04:15.374886 3337 ts_tablet_manager.cc:1397] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.000s
I20260924 16:04:15.374912 3340 log.cc:824] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:15.377918 3340 tablet_bootstrap.cc:492] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: No bootstrap required, opened a new log
I20260924 16:04:15.378217 3340 ts_tablet_manager.cc:1397] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 16:04:15.382673 3338 raft_consensus.cc:348] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } }
I20260924 16:04:15.383045 3338 raft_consensus.cc:374] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:15.383152 3338 raft_consensus.cc:729] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d85603163849a6ad2fac105650b358, State: Initialized, Role: FOLLOWER
I20260924 16:04:15.383751 3338 consensus_queue.cc:260] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } }
I20260924 16:04:15.384833 3048 heartbeater.cc:503] Master 127.16.233.190:41259 was elected leader, sending a full tablet report...
I20260924 16:04:15.385507 3338 ts_tablet_manager.cc:1428] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358: Time spent starting tablet: real 0.011s user 0.001s sys 0.009s
I20260924 16:04:15.385350 3337 raft_consensus.cc:348] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } }
I20260924 16:04:15.385753 3337 raft_consensus.cc:374] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:15.385845 3337 raft_consensus.cc:729] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604141e35cca4e03a7f5b59113b6398f, State: Initialized, Role: FOLLOWER
I20260924 16:04:15.386538 3337 consensus_queue.cc:260] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } }
I20260924 16:04:15.387650 3319 heartbeater.cc:503] Master 127.16.233.190:41259 was elected leader, sending a full tablet report...
I20260924 16:04:15.388567 3340 raft_consensus.cc:348] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } }
I20260924 16:04:15.389062 3340 raft_consensus.cc:374] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:15.389178 3340 raft_consensus.cc:729] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93ca95159084b989a41c826391c25d3, State: Initialized, Role: FOLLOWER
I20260924 16:04:15.389500 3337 ts_tablet_manager.cc:1428] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260924 16:04:15.389757 3340 consensus_queue.cc:260] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } }
I20260924 16:04:15.390925 3185 heartbeater.cc:503] Master 127.16.233.190:41259 was elected leader, sending a full tablet report...
I20260924 16:04:15.392509 3340 ts_tablet_manager.cc:1428] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260924 16:04:15.438248 3345 raft_consensus.cc:482] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:15.438716 3345 raft_consensus.cc:504] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } }
I20260924 16:04:15.440441 3345 leader_election.cc:288] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32d85603163849a6ad2fac105650b358 (127.16.233.129:36089), 604141e35cca4e03a7f5b59113b6398f (127.16.233.131:35343)
I20260924 16:04:15.446462 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.440780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:50243 (local address 127.16.233.129:36089)
0924 16:04:15.441038 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.441044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.441363 (+ 319us) server_negotiation.cc:407] Connection header received
0924 16:04:15.441605 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.441612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.441719 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.441815 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:15.442436 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.443124 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.444835 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.445560 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.445729 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.445798 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.446074 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.446199 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:04:15.446542 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.440620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.16.233.130:50243)
0924 16:04:15.441271 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.441292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:15.441450 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:15.441856 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:15.441861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:15.441905 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:15.442244 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.442254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.443245 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.443253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.444645 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.444656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.445736 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.445741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.445899 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.445921 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:04:15.446048 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 16:04:15.446171 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":312,"threads_started":1}
I20260924 16:04:15.447253 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.441477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:56257 (local address 127.16.233.131:35343)
0924 16:04:15.441716 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.441725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.442144 (+ 419us) server_negotiation.cc:407] Connection header received
0924 16:04:15.442370 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.442377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.442453 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.442547 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:15.443053 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.443729 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.445319 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.446260 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.446468 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.446541 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.446896 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.447024 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:04:15.447294 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.441271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:35343 (local address 127.16.233.130:56257)
0924 16:04:15.442042 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.442055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:15.442201 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:15.442561 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:15.442565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:15.442587 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:15.442895 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.442903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.443864 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.443870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.445145 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:15.445156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.446488 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.446495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:15.446723 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.446764 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:04:15.446870 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:04:15.446986 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":417,"threads_started":1}
I20260924 16:04:15.447665 3002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0891aede24294491a50acc9beb551313" candidate_uuid: "a93ca95159084b989a41c826391c25d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d85603163849a6ad2fac105650b358" is_pre_election: true
I20260924 16:04:15.448191 3002 raft_consensus.cc:2454] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a93ca95159084b989a41c826391c25d3 in term 0.
I20260924 16:04:15.448172 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0891aede24294491a50acc9beb551313" candidate_uuid: "a93ca95159084b989a41c826391c25d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604141e35cca4e03a7f5b59113b6398f" is_pre_election: true
I20260924 16:04:15.448683 3272 raft_consensus.cc:2454] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a93ca95159084b989a41c826391c25d3 in term 0.
I20260924 16:04:15.448879 3070 leader_election.cc:302] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d85603163849a6ad2fac105650b358, a93ca95159084b989a41c826391c25d3; no voters:
I20260924 16:04:15.449297 3345 raft_consensus.cc:2781] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:15.449458 3345 raft_consensus.cc:482] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:15.449579 3345 raft_consensus.cc:3037] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:15.451862 3345 raft_consensus.cc:504] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } }
I20260924 16:04:15.452431 3345 leader_election.cc:288] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [CANDIDATE]: Term 1 election: Requested vote from peers 32d85603163849a6ad2fac105650b358 (127.16.233.129:36089), 604141e35cca4e03a7f5b59113b6398f (127.16.233.131:35343)
I20260924 16:04:15.452939 3002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0891aede24294491a50acc9beb551313" candidate_uuid: "a93ca95159084b989a41c826391c25d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d85603163849a6ad2fac105650b358"
I20260924 16:04:15.452986 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0891aede24294491a50acc9beb551313" candidate_uuid: "a93ca95159084b989a41c826391c25d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604141e35cca4e03a7f5b59113b6398f"
I20260924 16:04:15.453150 3002 raft_consensus.cc:3037] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:15.453177 3272 raft_consensus.cc:3037] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:15.455597 3002 raft_consensus.cc:2454] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a93ca95159084b989a41c826391c25d3 in term 1.
I20260924 16:04:15.455693 3272 raft_consensus.cc:2454] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a93ca95159084b989a41c826391c25d3 in term 1.
I20260924 16:04:15.456095 3070 leader_election.cc:302] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d85603163849a6ad2fac105650b358, a93ca95159084b989a41c826391c25d3; no voters:
I20260924 16:04:15.456368 3345 raft_consensus.cc:2781] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:15.457005 3345 raft_consensus.cc:686] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 1 LEADER]: Becoming Leader. State: Replica: a93ca95159084b989a41c826391c25d3, State: Running, Role: LEADER
I20260924 16:04:15.457511 3345 consensus_queue.cc:237] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } }
I20260924 16:04:15.462087 2859 catalog_manager.cc:5495] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a93ca95159084b989a41c826391c25d3 (127.16.233.130). New cstate: current_term: 1 leader_uuid: "a93ca95159084b989a41c826391c25d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93ca95159084b989a41c826391c25d3" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 } health_report { overall_health: UNKNOWN } } }
I20260924 16:04:15.480100 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.474904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.16.233.130:36115)
0924 16:04:15.475101 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.475107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.475132 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:04:15.475214 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.475218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.475312 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.475471 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:15.475825 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.476595 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.478003 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.478482 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.478613 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.479051 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:15.479536 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:15.479692 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.479757 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.479846 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:04:15.488444 3272 raft_consensus.cc:1261] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [term 1 FOLLOWER]: Refusing update from remote peer a93ca95159084b989a41c826391c25d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:15.488437 3002 raft_consensus.cc:1261] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [term 1 FOLLOWER]: Refusing update from remote peer a93ca95159084b989a41c826391c25d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:15.489471 3348 consensus_queue.cc:1035] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d85603163849a6ad2fac105650b358" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 36089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:15.489799 3345 consensus_queue.cc:1035] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604141e35cca4e03a7f5b59113b6398f" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:15.501323 3351 mvcc.cc:205] Tried to move back new op lower bound from 7332928944069820416 to 7332928943958163456. Current Snapshot: MvccSnapshot[applied={T|T < 7332928944069820416}]
I20260924 16:04:15.521598 2860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:15.521926 2860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:15.529978 2860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c061932a7cf94241a78bbc3544fee8c9: Sending DeleteTablet for 3 replicas of tablet 0891aede24294491a50acc9beb551313
I20260924 16:04:15.531243 3116 tablet_service.cc:1514] Processing DeleteTablet for tablet 0891aede24294491a50acc9beb551313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:15 UTC) from {username='slave'} at 127.0.0.1:42876
I20260924 16:04:15.531584 2982 tablet_service.cc:1514] Processing DeleteTablet for tablet 0891aede24294491a50acc9beb551313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:15 UTC) from {username='slave'} at 127.0.0.1:37918
I20260924 16:04:15.532382 3362 tablet_replica.cc:330] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: stopping tablet replica
I20260924 16:04:15.533041 3362 raft_consensus.cc:2229] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:04:15.533815 3362 raft_consensus.cc:2258] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:15.533975 3252 tablet_service.cc:1514] Processing DeleteTablet for tablet 0891aede24294491a50acc9beb551313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:15 UTC) from {username='slave'} at 127.0.0.1:44814
I20260924 16:04:15.535082 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 2920
I20260924 16:04:15.542728 3362 ts_tablet_manager.cc:1905] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:15.552099 3362 ts_tablet_manager.cc:1918] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:15.562655 3362 log.cc:1196] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/0891aede24294491a50acc9beb551313
I20260924 16:04:15.563517 3362 ts_tablet_manager.cc:1939] T 0891aede24294491a50acc9beb551313 P a93ca95159084b989a41c826391c25d3: Deleting consensus metadata
I20260924 16:04:15.565722 2846 catalog_manager.cc:4841] TS a93ca95159084b989a41c826391c25d3 (127.16.233.130:36115): tablet 0891aede24294491a50acc9beb551313 (table test-table [id=3db4e289ff9849528b820b674e931001]) successfully deleted
I20260924 16:04:15.567350 3363 tablet_replica.cc:330] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358: stopping tablet replica
I20260924 16:04:15.567903 3363 raft_consensus.cc:2229] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:15.568480 3363 raft_consensus.cc:2258] T 0891aede24294491a50acc9beb551313 P 32d85603163849a6ad2fac105650b358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:15.569756 3364 tablet_replica.cc:330] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: stopping tablet replica
I20260924 16:04:15.570207 3364 raft_consensus.cc:2229] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:15.570767 3364 raft_consensus.cc:2258] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:15.573246 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.536693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.16.233.129:36089)
0924 16:04:15.536906 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.536914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.559439 (+ 22525us) server_negotiation.cc:407] Connection header received
0924 16:04:15.560043 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.560049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.560196 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.560278 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:15.561154 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.562525 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.564639 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.565264 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.565371 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.571272 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:15.571285 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:15.571306 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:15.572215 (+ 909us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:15.572290 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:15.572330 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:15.572441 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.572748 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.572883 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:04:15.574084 3043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:15.580525 3364 ts_tablet_manager.cc:1905] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:15.586752 3364 ts_tablet_manager.cc:1918] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:15.587100 3364 log.cc:1196] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/0891aede24294491a50acc9beb551313
I20260924 16:04:15.587813 3364 ts_tablet_manager.cc:1939] T 0891aede24294491a50acc9beb551313 P 604141e35cca4e03a7f5b59113b6398f: Deleting consensus metadata
I20260924 16:04:15.590494 2846 catalog_manager.cc:4841] TS 604141e35cca4e03a7f5b59113b6398f (127.16.233.131:35343): tablet 0891aede24294491a50acc9beb551313 (table test-table [id=3db4e289ff9849528b820b674e931001]) successfully deleted
I20260924 16:04:15.675657 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 2920
I20260924 16:04:15.700305 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 3052
W20260924 16:04:15.700304 2846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.129:36089 (error 108)
I20260924 16:04:15.703267 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.702853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:37946)
0924 16:04:15.703028 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.703133 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 16:04:15.703599 2846 catalog_manager.cc:4584] TS 32d85603163849a6ad2fac105650b358 (127.16.233.129:36089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0891aede24294491a50acc9beb551313: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36089: connect: Connection refused (error 111)
I20260924 16:04:15.708654 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.701261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.16.233.130:36115)
0924 16:04:15.701467 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.701474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.701575 (+ 101us) server_negotiation.cc:407] Connection header received
0924 16:04:15.701819 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.701828 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.701939 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.702063 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:15.702768 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.703870 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.705184 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.705785 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.705914 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.707007 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:15.707019 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:15.707036 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:15.707859 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:15.707920 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:15.707946 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:15.708066 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.708148 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.708254 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:04:15.709378 3180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:15.757213 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.756647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:37954)
0924 16:04:15.756878 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.757000 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:04:15.758262 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.757881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:37958)
0924 16:04:15.758040 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.758124 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 16:04:15.818866 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.818361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:37960)
0924 16:04:15.818603 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.818687 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:04:15.819949 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.819601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:37976)
0924 16:04:15.819751 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.819826 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 16:04:15.831763 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 3052
I20260924 16:04:15.850467 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 3188
I20260924 16:04:15.869252 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.851241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.16.233.131:35343)
0924 16:04:15.851416 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:15.851421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:15.851439 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:04:15.851591 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:15.851595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:15.851675 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:15.851756 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:15.852131 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.852901 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.854165 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:15.854756 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:15.854851 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:15.867939 (+ 13088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:15.867951 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:15.867966 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:15.868566 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:15.868623 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:15.868649 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:15.868755 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:15.868878 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 16:04:15.868998 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:04:15.869916 3313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:15.900812 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.900234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:37982)
0924 16:04:15.900504 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.900623 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 16:04:15.901964 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:15.901488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:37992)
0924 16:04:15.901715 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:15.901837 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 16:04:15.959617 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 3188
I20260924 16:04:15.976748 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 2828
I20260924 16:04:15.977830 2890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:16.086752 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:16.085919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:36089 (local address 127.0.0.1:37994)
0924 16:04:16.086160 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:16.086271 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 16:04:16.086830 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 2828
2026-09-24T16:04:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2774 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:16 dist-test-slave-xn4h krb5kdc[3373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:16 dist-test-slave-xn4h krb5kdc[3373](info): set up 2 sockets
Sep 24 16:04:16 dist-test-slave-xn4h krb5kdc[3373](info): commencing operation
krb5kdc: starting...
W20260924 16:04:17.199925 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:17 dist-test-slave-xn4h krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:04:17.437973 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:39375
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:39375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:17.764672 3386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:17.765017 3386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:17.765120 3386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:17.772270 3386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:17.772374 3386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:17.772423 3386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:17.772481 3386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:17.772526 3386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:17.782259 3386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:39375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:17.783562 3386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:17.784824 3386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:17.796113 3392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:17.796274 3391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:17.796245 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:17.797327 3386 server_base.cc:1034] running on GCE node
I20260924 16:04:17.798156 3386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:17.799523 3386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:17.800735 3386 hybrid_clock.cc:648] HybridClock initialized: now 1790265857800695 us; error 33 us; skew 500 ppm
I20260924 16:04:17.801077 3386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:17 dist-test-slave-xn4h krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265857, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:17.804112 3386 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:04:17.805521 3386 webserver.cc:460] Webserver started at http://127.16.233.190:43225/ using document root <none> and password file <none>
I20260924 16:04:17.806202 3386 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:17.806370 3386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:17.806699 3386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:17.809617 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb"
format_stamp: "Formatted at 2026-09-24 16:04:17 on dist-test-slave-xn4h"
server_key: "8d009dc1a4c83d29d3891a9f65361538"
server_key_iv: "6a48231aeeb2febe4875d645c0082758"
server_key_version: "encryptionkey@0"
I20260924 16:04:17.810427 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb"
format_stamp: "Formatted at 2026-09-24 16:04:17 on dist-test-slave-xn4h"
server_key: "8d009dc1a4c83d29d3891a9f65361538"
server_key_iv: "6a48231aeeb2febe4875d645c0082758"
server_key_version: "encryptionkey@0"
I20260924 16:04:17.815356 3386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 16:04:17.818560 3401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:17.819837 3386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:04:17.820032 3386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb"
format_stamp: "Formatted at 2026-09-24 16:04:17 on dist-test-slave-xn4h"
server_key: "8d009dc1a4c83d29d3891a9f65361538"
server_key_iv: "6a48231aeeb2febe4875d645c0082758"
server_key_version: "encryptionkey@0"
I20260924 16:04:17.820227 3386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:17.830680 3386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:17.834249 3386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:17.834613 3386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:17.852996 3386 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:39375
I20260924 16:04:17.853019 3453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:39375 every 8 connection(s)
I20260924 16:04:17.854432 3386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:17.858155 3454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:17.861464 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 3386
I20260924 16:04:17.861652 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:17.862248 17318 external_mini_cluster.cc:1428] Setting key a72ab7eb8ee21703f9a330b54f1c3f12
I20260924 16:04:17.870814 3454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb: Bootstrap starting.
I20260924 16:04:17.874668 3454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:17.875983 3454 log.cc:824] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:04:17 dist-test-slave-xn4h krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:17.878892 3454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb: No bootstrap required, opened a new log
I20260924 16:04:17.886924 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:17.864164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.16.233.190:39375)
0924 16:04:17.864904 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:17.864916 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:17.864955 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:04:17.866657 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:17.866680 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:17.867090 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:17.867403 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:17.869273 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:17.870630 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:17.872436 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:17.873109 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:17.873236 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:17.878393 (+ 5157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:17.878403 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:17.878416 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:17.878449 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:17.882386 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:17.883093 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:17.883103 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:17.883111 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:17.883325 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:17.883717 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:17.883722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:17.883726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:17.884391 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:17.884697 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:17.885067 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 16:04:17.885527 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":342,"threads_started":1}
I20260924 16:04:17.889261 3454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39375 } }
I20260924 16:04:17.889647 3454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:17.889783 3454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32766bc18d6a4f9ea9a15f3ee3afe0fb, State: Initialized, Role: FOLLOWER
I20260924 16:04:17.890401 3454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39375 } }
I20260924 16:04:17.890635 3454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:17.890774 3454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:17.890952 3454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:17.893044 3454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39375 } }
I20260924 16:04:17.893512 3454 leader_election.cc:302] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32766bc18d6a4f9ea9a15f3ee3afe0fb; no voters:
I20260924 16:04:17.894069 3454 leader_election.cc:288] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:17.894207 3459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:17.894973 3459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [term 1 LEADER]: Becoming Leader. State: Replica: 32766bc18d6a4f9ea9a15f3ee3afe0fb, State: Running, Role: LEADER
I20260924 16:04:17.895499 3459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39375 } }
I20260924 16:04:17.896111 3454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:17.898337 3460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39375 } } }
I20260924 16:04:17.898397 3461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32766bc18d6a4f9ea9a15f3ee3afe0fb. Latest consensus state: current_term: 1 leader_uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32766bc18d6a4f9ea9a15f3ee3afe0fb" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39375 } } }
I20260924 16:04:17.898797 3461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:17.898797 3460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:17.903600 3466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:17.909796 3466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:17.918995 3466 catalog_manager.cc:1347] Generated new cluster ID: 9a2e87bf07994eceaf987792fdea1c56
I20260924 16:04:17.919107 3466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:17.945586 3466 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:17.946985 3466 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:17.953672 3466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb: Generated new TSK 0
I20260924 16:04:17.954583 3466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 16:04:18.017751 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:39375
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:18.375455 3481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:18.375743 3481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:18.375834 3481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:18.383597 3481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:18.383718 3481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:18.383878 3481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:04:18.393384 3481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:39375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:18.394805 3481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:18.396189 3481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:18.409700 3486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:18.409773 3487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:18.409991 3489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:18.410806 3481 server_base.cc:1034] running on GCE node
I20260924 16:04:18.411487 3481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:18.412468 3481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:18.413663 3481 hybrid_clock.cc:648] HybridClock initialized: now 1790265858413624 us; error 35 us; skew 500 ppm
I20260924 16:04:18.413968 3481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:18 dist-test-slave-xn4h krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265858, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:18.417021 3481 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:04:18.418617 3481 webserver.cc:460] Webserver started at http://127.16.233.129:37083/ using document root <none> and password file <none>
I20260924 16:04:18.419297 3481 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:18.419448 3481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:18.419780 3481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:18.422751 3481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "5526a7f538654b5792958c175cf3ccd0"
format_stamp: "Formatted at 2026-09-24 16:04:18 on dist-test-slave-xn4h"
server_key: "9dab9f15cc0322f4513aa8856c5006dd"
server_key_iv: "421ecd038cbe715e2c2389769cd4d0de"
server_key_version: "encryptionkey@0"
I20260924 16:04:18.423560 3481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "5526a7f538654b5792958c175cf3ccd0"
format_stamp: "Formatted at 2026-09-24 16:04:18 on dist-test-slave-xn4h"
server_key: "9dab9f15cc0322f4513aa8856c5006dd"
server_key_iv: "421ecd038cbe715e2c2389769cd4d0de"
server_key_version: "encryptionkey@0"
I20260924 16:04:18.428872 3481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:04:18.432382 3496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:18.433689 3481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:04:18.433866 3481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "5526a7f538654b5792958c175cf3ccd0"
format_stamp: "Formatted at 2026-09-24 16:04:18 on dist-test-slave-xn4h"
server_key: "9dab9f15cc0322f4513aa8856c5006dd"
server_key_iv: "421ecd038cbe715e2c2389769cd4d0de"
server_key_version: "encryptionkey@0"
I20260924 16:04:18.434070 3481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:18.446558 3481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:18.450307 3481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:18.450649 3481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:18.451560 3481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:18.452904 3481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:18.452978 3481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:18.453135 3481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:18.453194 3481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:18.486132 3481 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:41373
I20260924 16:04:18.486177 3611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:41373 every 8 connection(s)
I20260924 16:04:18.487530 3481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:18.496739 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 3481
I20260924 16:04:18.496964 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:04:18.497478 17318 external_mini_cluster.cc:1428] Setting key b781b53fe62908de7b1082af467a2cf7
Sep 24 16:04:18 dist-test-slave-xn4h krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265858, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:18.509171 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:18.490251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:38789 (local address 127.16.233.190:39375)
0924 16:04:18.490754 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:18.490759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:18.491592 (+ 833us) server_negotiation.cc:407] Connection header received
0924 16:04:18.493467 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:18.493472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:18.493562 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:18.493683 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:18.495259 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:18.496016 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:18.497785 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:18.498381 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:18.498529 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:18.502943 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:18.502948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:18.502951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:18.502981 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:18.505725 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:18.506682 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:18.506686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:18.506689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:18.506804 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:18.507289 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:18.507292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:18.507293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:18.507542 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:18.507716 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:18.508420 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 16:04:18.508693 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":259,"threads_started":1}
I20260924 16:04:18.509835 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:18.490720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:39375 (local address 127.16.233.129:38789)
0924 16:04:18.491423 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:18.491468 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:18.493194 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:18.493916 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:18.493930 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:18.494507 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:18.495090 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:18.495108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:18.496179 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:18.496183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:18.497640 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:18.497652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:18.499175 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:18.499180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:18.499369 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:18.500063 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:18.500084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:18.502733 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:18.505859 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:18.505870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:18.505890 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:18.506569 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:18.506911 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:18.506918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:18.506925 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:18.507174 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:18.507703 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:18.507709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:18.508028 (+ 319us) client_negotiation.cc:769] Sending connection context
0924 16:04:18.508249 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 16:04:18.508631 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":263,"threads_started":1}
I20260924 16:04:18.511430 3612 heartbeater.cc:347] Connected to a master server at 127.16.233.190:39375
I20260924 16:04:18.511756 3612 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:18.512480 3612 heartbeater.cc:511] Master 127.16.233.190:39375 requested a full tablet report, sending...
I20260924 16:04:18.514577 3418 ts_manager.cc:194] Registered new tserver with Master: 5526a7f538654b5792958c175cf3ccd0 (127.16.233.129:41373)
I20260924 16:04:18.515987 3418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:38789
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 16:04:18.553849 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:39375
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:18.879256 3618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:18.879528 3618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:18.879694 3618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:18.887081 3618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:18.887202 3618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:18.887391 3618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:04:18.898001 3618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:39375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:18.899381 3618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:18.900740 3618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:18.913968 3623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:18.914005 3624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:18.914064 3626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:18.914991 3618 server_base.cc:1034] running on GCE node
I20260924 16:04:18.915584 3618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:18.916466 3618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:18.917923 3618 hybrid_clock.cc:648] HybridClock initialized: now 1790265858917872 us; error 44 us; skew 500 ppm
I20260924 16:04:18.918264 3618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:18 dist-test-slave-xn4h krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265858, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:18.921466 3618 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:04:18.923228 3618 webserver.cc:460] Webserver started at http://127.16.233.130:44323/ using document root <none> and password file <none>
I20260924 16:04:18.924258 3618 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:18.924401 3618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:18.924793 3618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:18.928205 3618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "537690d6912c4b40a3b0d389df3487ff"
format_stamp: "Formatted at 2026-09-24 16:04:18 on dist-test-slave-xn4h"
server_key: "5f34bc2449f4ac78af610f2109fcc991"
server_key_iv: "ba7ad7a8941b43f7234f7c68d6428a89"
server_key_version: "encryptionkey@0"
I20260924 16:04:18.929008 3618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "537690d6912c4b40a3b0d389df3487ff"
format_stamp: "Formatted at 2026-09-24 16:04:18 on dist-test-slave-xn4h"
server_key: "5f34bc2449f4ac78af610f2109fcc991"
server_key_iv: "ba7ad7a8941b43f7234f7c68d6428a89"
server_key_version: "encryptionkey@0"
I20260924 16:04:18.934774 3618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:04:18.938445 3633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:18.939869 3618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:04:18.940073 3618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "537690d6912c4b40a3b0d389df3487ff"
format_stamp: "Formatted at 2026-09-24 16:04:18 on dist-test-slave-xn4h"
server_key: "5f34bc2449f4ac78af610f2109fcc991"
server_key_iv: "ba7ad7a8941b43f7234f7c68d6428a89"
server_key_version: "encryptionkey@0"
I20260924 16:04:18.940294 3618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:18.967656 3618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:18.972256 3618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:18.972568 3618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:18.973701 3618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:18.975208 3618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:18.975311 3618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:18.975468 3618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:18.975543 3618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:19.009068 3618 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:46489
I20260924 16:04:19.009120 3746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:46489 every 8 connection(s)
I20260924 16:04:19.010435 3618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:19.018476 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 3618
I20260924 16:04:19.018676 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:04:19.019126 17318 external_mini_cluster.cc:1428] Setting key 751e960e63de8652854b250b23d6e3bb
Sep 24 16:04:19 dist-test-slave-xn4h krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265858, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:19.031092 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.013013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:59855 (local address 127.16.233.190:39375)
0924 16:04:19.013448 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.013452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.014436 (+ 984us) server_negotiation.cc:407] Connection header received
0924 16:04:19.016234 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.016240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.016329 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.016443 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:19.017918 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.018796 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.020561 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.021165 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.021302 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.025178 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:19.025183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:19.025187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:19.025221 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:19.027891 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:19.028665 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:19.028672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:19.028677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:19.028814 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:19.029249 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:19.029257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:19.029260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:19.029554 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:19.029710 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.030471 (+ 761us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.030701 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":217,"threads_started":1}
I20260924 16:04:19.032179 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.013571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:39375 (local address 127.16.233.130:59855)
0924 16:04:19.014229 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:19.014271 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:19.015974 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:19.016681 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:19.016689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:19.017184 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:19.017747 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.017765 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.018932 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.018941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.020408 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.020432 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.021390 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.021398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.021635 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.022374 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:19.022398 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:19.024949 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:19.028055 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:19.028065 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:19.028084 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:19.028540 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:19.028927 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:19.028933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:19.028937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:19.029132 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:19.029686 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:19.029694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:19.030052 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 16:04:19.030394 (+ 342us) client_negotiation.cc:240] Negotiation successful
0924 16:04:19.030953 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":251,"threads_started":1}
I20260924 16:04:19.033759 3747 heartbeater.cc:347] Connected to a master server at 127.16.233.190:39375
I20260924 16:04:19.034073 3747 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:19.034844 3747 heartbeater.cc:511] Master 127.16.233.190:39375 requested a full tablet report, sending...
I20260924 16:04:19.036329 3418 ts_manager.cc:194] Registered new tserver with Master: 537690d6912c4b40a3b0d389df3487ff (127.16.233.130:46489)
I20260924 16:04:19.037618 3418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:59855
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 16:04:19.073832 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:39375
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:43337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:19.404160 3754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:19.404500 3754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:19.404681 3754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:19.412027 3754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:19.412143 3754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:19.412326 3754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:04:19.422281 3754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:43337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:39375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:19.423731 3754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:19.425150 3754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:19.438864 3760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:19.438934 3759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:19.438946 3762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:19.439733 3754 server_base.cc:1034] running on GCE node
I20260924 16:04:19.440387 3754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:19.441318 3754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:19.442525 3754 hybrid_clock.cc:648] HybridClock initialized: now 1790265859442449 us; error 56 us; skew 500 ppm
I20260924 16:04:19.442878 3754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:19 dist-test-slave-xn4h krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265859, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:19.446067 3754 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:04:19.447666 3754 webserver.cc:460] Webserver started at http://127.16.233.131:36667/ using document root <none> and password file <none>
I20260924 16:04:19.448383 3754 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:19.448529 3754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:19.448858 3754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:19.451819 3754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "91432a6e6145433db46ed4a79f4ef964"
format_stamp: "Formatted at 2026-09-24 16:04:19 on dist-test-slave-xn4h"
server_key: "7362c1cae5455e2ea0557de843f0e832"
server_key_iv: "bcd27755b32031dfb0c88e27de26ae84"
server_key_version: "encryptionkey@0"
I20260924 16:04:19.452587 3754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "91432a6e6145433db46ed4a79f4ef964"
format_stamp: "Formatted at 2026-09-24 16:04:19 on dist-test-slave-xn4h"
server_key: "7362c1cae5455e2ea0557de843f0e832"
server_key_iv: "bcd27755b32031dfb0c88e27de26ae84"
server_key_version: "encryptionkey@0"
I20260924 16:04:19.457945 3754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:04:19.461490 3769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:19.462937 3754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:19.463119 3754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "91432a6e6145433db46ed4a79f4ef964"
format_stamp: "Formatted at 2026-09-24 16:04:19 on dist-test-slave-xn4h"
server_key: "7362c1cae5455e2ea0557de843f0e832"
server_key_iv: "bcd27755b32031dfb0c88e27de26ae84"
server_key_version: "encryptionkey@0"
I20260924 16:04:19.463326 3754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:19.476927 3754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:19.481087 3754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:19.481388 3754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:19.482415 3754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:19.483776 3754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:19.483850 3754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:19.484004 3754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:19.484056 3754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:19.517515 3754 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:38773
I20260924 16:04:19.517558 3882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:38773 every 8 connection(s)
I20260924 16:04:19.518852 3612 heartbeater.cc:503] Master 127.16.233.190:39375 was elected leader, sending a full tablet report...
I20260924 16:04:19.519127 3754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:19.528561 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 3754
I20260924 16:04:19.528764 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:04:19.529313 17318 external_mini_cluster.cc:1428] Setting key 5948ebe0cf6f74048a7f57c269dac218
Sep 24 16:04:19 dist-test-slave-xn4h krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265859, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:19.540479 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.521991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:38473 (local address 127.16.233.190:39375)
0924 16:04:19.522192 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.522199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.523640 (+ 1441us) server_negotiation.cc:407] Connection header received
0924 16:04:19.525612 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.525616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.525685 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.525776 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:19.527356 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.528105 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.530437 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.530882 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.530994 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.534784 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:19.534788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:19.534791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:19.534823 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:19.537215 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:19.538023 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:19.538027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:19.538029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:19.538150 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:19.538655 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:19.538660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:19.538664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:19.539008 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:19.539167 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.539896 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.540139 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:04:19.541656 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.522703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:39375 (local address 127.16.233.131:38473)
0924 16:04:19.523472 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:19.523539 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:19.525277 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:19.525976 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:19.525987 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:19.526603 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:19.527206 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.527224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.528259 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.528266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.530232 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.530249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.531118 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.531124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.531441 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.532229 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:19.532259 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:19.534573 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:19.537358 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:19.537367 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:19.537385 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:19.537895 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:19.538257 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:19.538262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:19.538266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:19.538534 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:19.539133 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:19.539140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:19.539501 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 16:04:19.539767 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 16:04:19.540340 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":269,"threads_started":1}
I20260924 16:04:19.543437 3883 heartbeater.cc:347] Connected to a master server at 127.16.233.190:39375
I20260924 16:04:19.544102 3883 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:19.545017 3883 heartbeater.cc:511] Master 127.16.233.190:39375 requested a full tablet report, sending...
I20260924 16:04:19.546535 3418 ts_manager.cc:194] Registered new tserver with Master: 91432a6e6145433db46ed4a79f4ef964 (127.16.233.131:38773)
I20260924 16:04:19.547355 3418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:38473
I20260924 16:04:19.557331 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:19.574608 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.561611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.16.233.190:39375)
0924 16:04:19.561780 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.561784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.562255 (+ 471us) server_negotiation.cc:407] Connection header received
0924 16:04:19.562426 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.562430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.562498 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.562587 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:19.563978 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.564703 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.566002 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.566496 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.566626 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.569325 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:19.569332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:19.569337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:19.569374 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:19.572015 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:19.572807 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:19.572811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:19.572813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:19.572971 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:19.573342 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:19.573345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:19.573347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:19.573626 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:19.573827 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.574147 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.574280 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:04:19.578769 3418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:19.585613 3418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:19.619962 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.598388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.16.233.130:46489)
0924 16:04:19.609177 (+ 10789us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.609192 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.609234 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:04:19.609379 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.609392 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.609753 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.612347 (+ 2594us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:19.612676 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.614341 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.617399 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.618743 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.618929 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.619119 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.619442 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.619610 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10539,"thread_start_us":11565,"threads_started":1}
I20260924 16:04:19.620071 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.597950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:46489 (local address 127.0.0.1:55280)
0924 16:04:19.600055 (+ 2105us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:19.600104 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:19.600279 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:19.610365 (+ 10086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:19.610377 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:19.610424 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:19.612046 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.612066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.614519 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.614524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.617229 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.617243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.618969 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.618976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.619189 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.619226 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:04:19.619392 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 16:04:19.619585 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1837,"mutex_wait_us":2,"thread_start_us":15690,"threads_started":1}
I20260924 16:04:19.622107 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.597966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:41373 (local address 127.0.0.1:51508)
0924 16:04:19.613926 (+ 15960us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:19.613949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:19.614074 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:19.614894 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:19.614899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:19.614932 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:19.615702 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.615716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.617545 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.617553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.619400 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.619415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.621245 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.621252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.621439 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.621470 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:04:19.621568 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:04:19.621704 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15843,"mutex_wait_us":161,"thread_start_us":1537,"threads_started":1}
I20260924 16:04:19.622112 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.598087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.16.233.129:41373)
0924 16:04:19.613317 (+ 15230us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.613326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.614227 (+ 901us) server_negotiation.cc:407] Connection header received
0924 16:04:19.614379 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.614388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.614662 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.615335 (+ 673us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:19.615910 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.617422 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.619597 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.621035 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.621267 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.621437 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.621621 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.621781 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15037,"thread_start_us":15231,"threads_started":1}
I20260924 16:04:19.624445 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.600458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:38773 (local address 127.0.0.1:50162)
0924 16:04:19.615010 (+ 14552us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:19.615028 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:19.615187 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:19.616076 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:19.616081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:19.616103 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:19.616518 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.616531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.618340 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.618346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.620824 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.620839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.623661 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.623668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.623836 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.623870 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:04:19.623964 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:04:19.624093 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14429,"thread_start_us":14267,"threads_started":1}
I20260924 16:04:19.624348 3682 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5bccaa61654268a52821feff8f53ca (DEFAULT_TABLE table=test-table [id=01946c241a6040a08ca91d8059b4195f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:19.625070 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.600819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.16.233.131:38773)
0924 16:04:19.614475 (+ 13656us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.614487 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.615147 (+ 660us) server_negotiation.cc:407] Connection header received
0924 16:04:19.615359 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.615368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.615608 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.615793 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:19.616765 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.618164 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.620993 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.623408 (+ 2415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.623664 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.623824 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.624316 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.624486 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13407,"thread_start_us":13246,"threads_started":1}
I20260924 16:04:19.625841 3682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5bccaa61654268a52821feff8f53ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:19.626036 3545 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5bccaa61654268a52821feff8f53ca (DEFAULT_TABLE table=test-table [id=01946c241a6040a08ca91d8059b4195f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:19.627283 3545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5bccaa61654268a52821feff8f53ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:19.628737 3817 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5bccaa61654268a52821feff8f53ca (DEFAULT_TABLE table=test-table [id=01946c241a6040a08ca91d8059b4195f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:19.630086 3817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5bccaa61654268a52821feff8f53ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:19.640265 3902 tablet_bootstrap.cc:492] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: Bootstrap starting.
I20260924 16:04:19.643532 3902 tablet_bootstrap.cc:654] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:19.644685 3902 log.cc:824] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:19.646203 3903 tablet_bootstrap.cc:492] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: Bootstrap starting.
I20260924 16:04:19.648085 3905 tablet_bootstrap.cc:492] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: Bootstrap starting.
I20260924 16:04:19.648300 3902 tablet_bootstrap.cc:492] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: No bootstrap required, opened a new log
I20260924 16:04:19.648638 3902 ts_tablet_manager.cc:1397] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260924 16:04:19.649878 3903 tablet_bootstrap.cc:654] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:19.651225 3903 log.cc:824] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:19.651319 3905 tablet_bootstrap.cc:654] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:19.652565 3905 log.cc:824] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:19.654589 3903 tablet_bootstrap.cc:492] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: No bootstrap required, opened a new log
I20260924 16:04:19.654881 3903 ts_tablet_manager.cc:1397] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 16:04:19.655207 3905 tablet_bootstrap.cc:492] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: No bootstrap required, opened a new log
I20260924 16:04:19.655460 3905 ts_tablet_manager.cc:1397] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 16:04:19.659555 3902 raft_consensus.cc:348] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } }
I20260924 16:04:19.660070 3902 raft_consensus.cc:374] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:19.660172 3902 raft_consensus.cc:729] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5526a7f538654b5792958c175cf3ccd0, State: Initialized, Role: FOLLOWER
I20260924 16:04:19.660835 3902 consensus_queue.cc:260] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } }
I20260924 16:04:19.662719 3902 ts_tablet_manager.cc:1428] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260924 16:04:19.663592 3905 raft_consensus.cc:348] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } }
I20260924 16:04:19.664013 3905 raft_consensus.cc:374] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:19.664105 3905 raft_consensus.cc:729] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91432a6e6145433db46ed4a79f4ef964, State: Initialized, Role: FOLLOWER
I20260924 16:04:19.664717 3905 consensus_queue.cc:260] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } }
I20260924 16:04:19.665196 3903 raft_consensus.cc:348] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } }
I20260924 16:04:19.665621 3903 raft_consensus.cc:374] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:19.665746 3903 raft_consensus.cc:729] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537690d6912c4b40a3b0d389df3487ff, State: Initialized, Role: FOLLOWER
I20260924 16:04:19.665946 3883 heartbeater.cc:503] Master 127.16.233.190:39375 was elected leader, sending a full tablet report...
I20260924 16:04:19.666329 3903 consensus_queue.cc:260] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } }
I20260924 16:04:19.666499 3905 ts_tablet_manager.cc:1428] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 16:04:19.667645 3747 heartbeater.cc:503] Master 127.16.233.190:39375 was elected leader, sending a full tablet report...
I20260924 16:04:19.668908 3903 ts_tablet_manager.cc:1428] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260924 16:04:19.740084 3613 tablet.cc:2369] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:04:19.762548 3748 tablet.cc:2369] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:04:19.765332 3908 raft_consensus.cc:482] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:19.765657 3908 raft_consensus.cc:504] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } }
I20260924 16:04:19.767426 3908 leader_election.cc:288] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 537690d6912c4b40a3b0d389df3487ff (127.16.233.130:46489), 91432a6e6145433db46ed4a79f4ef964 (127.16.233.131:38773)
W20260924 16:04:19.770974 3884 tablet.cc:2369] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:04:19.774515 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.767807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:48595 (local address 127.16.233.130:46489)
0924 16:04:19.768026 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.768036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.768716 (+ 680us) server_negotiation.cc:407] Connection header received
0924 16:04:19.768891 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.768898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.768982 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.769074 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:19.769729 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.770591 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.772540 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.773508 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.773625 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.773676 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.774079 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.774213 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:04:19.774626 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.767766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:46489 (local address 127.16.233.129:48595)
0924 16:04:19.768601 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:19.768617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:19.768743 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:19.769108 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:19.769112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:19.769139 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:19.769576 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.769586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.770732 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.770737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.772387 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.772399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.773719 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.773726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.773904 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.773939 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:04:19.774040 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 16:04:19.774179 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":23,"thread_start_us":469,"threads_started":1}
I20260924 16:04:19.774919 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.767909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:46089 (local address 127.16.233.131:38773)
0924 16:04:19.768163 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.768172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.768928 (+ 756us) server_negotiation.cc:407] Connection header received
0924 16:04:19.769113 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.769118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.769198 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.769306 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:19.769972 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.770824 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.773044 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.773934 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.774095 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.774160 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.774527 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.774674 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 16:04:19.775010 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.767766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:38773 (local address 127.16.233.129:46089)
0924 16:04:19.768815 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:19.768834 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:19.768963 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:19.769336 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:19.769340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:19.769359 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:19.769803 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.769816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.770986 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.770992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.772884 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:19.772899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.774165 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.774171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:19.774355 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.774390 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:04:19.774489 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 16:04:19.774641 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":62,"thread_start_us":603,"threads_started":1}
I20260924 16:04:19.775568 3702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5bccaa61654268a52821feff8f53ca" candidate_uuid: "5526a7f538654b5792958c175cf3ccd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "537690d6912c4b40a3b0d389df3487ff" is_pre_election: true
I20260924 16:04:19.775913 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5bccaa61654268a52821feff8f53ca" candidate_uuid: "5526a7f538654b5792958c175cf3ccd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91432a6e6145433db46ed4a79f4ef964" is_pre_election: true
I20260924 16:04:19.776063 3702 raft_consensus.cc:2454] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5526a7f538654b5792958c175cf3ccd0 in term 0.
I20260924 16:04:19.776397 3837 raft_consensus.cc:2454] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5526a7f538654b5792958c175cf3ccd0 in term 0.
I20260924 16:04:19.776729 3497 leader_election.cc:302] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 537690d6912c4b40a3b0d389df3487ff, 5526a7f538654b5792958c175cf3ccd0; no voters:
I20260924 16:04:19.777137 3908 raft_consensus.cc:2781] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:19.777279 3908 raft_consensus.cc:482] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:19.777397 3908 raft_consensus.cc:3037] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:19.779656 3908 raft_consensus.cc:504] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } }
I20260924 16:04:19.780292 3908 leader_election.cc:288] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [CANDIDATE]: Term 1 election: Requested vote from peers 537690d6912c4b40a3b0d389df3487ff (127.16.233.130:46489), 91432a6e6145433db46ed4a79f4ef964 (127.16.233.131:38773)
I20260924 16:04:19.780731 3702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5bccaa61654268a52821feff8f53ca" candidate_uuid: "5526a7f538654b5792958c175cf3ccd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "537690d6912c4b40a3b0d389df3487ff"
I20260924 16:04:19.780797 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5bccaa61654268a52821feff8f53ca" candidate_uuid: "5526a7f538654b5792958c175cf3ccd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91432a6e6145433db46ed4a79f4ef964"
I20260924 16:04:19.780920 3702 raft_consensus.cc:3037] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:19.781052 3837 raft_consensus.cc:3037] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:19.783179 3702 raft_consensus.cc:2454] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5526a7f538654b5792958c175cf3ccd0 in term 1.
I20260924 16:04:19.783388 3837 raft_consensus.cc:2454] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5526a7f538654b5792958c175cf3ccd0 in term 1.
I20260924 16:04:19.783609 3497 leader_election.cc:302] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 537690d6912c4b40a3b0d389df3487ff, 5526a7f538654b5792958c175cf3ccd0; no voters:
I20260924 16:04:19.783906 3908 raft_consensus.cc:2781] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:19.784322 3908 raft_consensus.cc:686] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 1 LEADER]: Becoming Leader. State: Replica: 5526a7f538654b5792958c175cf3ccd0, State: Running, Role: LEADER
I20260924 16:04:19.784797 3908 consensus_queue.cc:237] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } }
I20260924 16:04:19.789152 3418 catalog_manager.cc:5495] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5526a7f538654b5792958c175cf3ccd0 (127.16.233.129). New cstate: current_term: 1 leader_uuid: "5526a7f538654b5792958c175cf3ccd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526a7f538654b5792958c175cf3ccd0" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 } health_report { overall_health: UNKNOWN } } }
I20260924 16:04:19.849771 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.844603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.16.233.129:41373)
0924 16:04:19.844826 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.844831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.844848 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:04:19.844998 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.845002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.845098 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.845218 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:19.845579 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.846267 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.847692 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.848075 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.848164 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.848651 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:19.849142 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:19.849270 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.849432 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.849496 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:04:19.858774 3837 raft_consensus.cc:1261] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [term 1 FOLLOWER]: Refusing update from remote peer 5526a7f538654b5792958c175cf3ccd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:19.858772 3702 raft_consensus.cc:1261] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [term 1 FOLLOWER]: Refusing update from remote peer 5526a7f538654b5792958c175cf3ccd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:19.859956 3913 consensus_queue.cc:1035] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91432a6e6145433db46ed4a79f4ef964" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:19.860414 3908 consensus_queue.cc:1035] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "537690d6912c4b40a3b0d389df3487ff" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:19.883795 3918 mvcc.cc:205] Tried to move back new op lower bound from 7332928961970425856 to 7332928961682112512. Current Snapshot: MvccSnapshot[applied={T|T < 7332928961970425856}]
I20260924 16:04:19.891388 3418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:19.891659 3418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:19.895306 3418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32766bc18d6a4f9ea9a15f3ee3afe0fb: Sending DeleteTablet for 3 replicas of tablet 0a5bccaa61654268a52821feff8f53ca
I20260924 16:04:19.896548 3817 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5bccaa61654268a52821feff8f53ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:19 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:50162
I20260924 16:04:19.896548 3682 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5bccaa61654268a52821feff8f53ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:19 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:55280
I20260924 16:04:19.897253 3545 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5bccaa61654268a52821feff8f53ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:19 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:51508
I20260924 16:04:19.897781 3927 tablet_replica.cc:330] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: stopping tablet replica
I20260924 16:04:19.898259 3927 raft_consensus.cc:2229] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:19.898855 3927 raft_consensus.cc:2258] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:19.899899 3928 tablet_replica.cc:330] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: stopping tablet replica
I20260924 16:04:19.900374 3928 raft_consensus.cc:2229] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:19.900630 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 3481
I20260924 16:04:19.900974 3928 raft_consensus.cc:2258] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:19.903287 3928 ts_tablet_manager.cc:1905] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:19.908840 3929 tablet_replica.cc:330] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: stopping tablet replica
I20260924 16:04:19.909376 3929 raft_consensus.cc:2229] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:04:19.910094 3929 raft_consensus.cc:2258] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:19.910717 3928 ts_tablet_manager.cc:1918] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:19.910948 3928 log.cc:1196] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/0a5bccaa61654268a52821feff8f53ca
I20260924 16:04:19.911577 3928 ts_tablet_manager.cc:1939] T 0a5bccaa61654268a52821feff8f53ca P 91432a6e6145433db46ed4a79f4ef964: Deleting consensus metadata
I20260924 16:04:19.911867 3927 ts_tablet_manager.cc:1905] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:19.912357 3929 ts_tablet_manager.cc:1905] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:19.913794 3405 catalog_manager.cc:4841] TS 91432a6e6145433db46ed4a79f4ef964 (127.16.233.131:38773): tablet 0a5bccaa61654268a52821feff8f53ca (table test-table [id=01946c241a6040a08ca91d8059b4195f]) successfully deleted
I20260924 16:04:19.918874 3927 ts_tablet_manager.cc:1918] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:19.919050 3927 log.cc:1196] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/0a5bccaa61654268a52821feff8f53ca
I20260924 16:04:19.919577 3927 ts_tablet_manager.cc:1939] T 0a5bccaa61654268a52821feff8f53ca P 537690d6912c4b40a3b0d389df3487ff: Deleting consensus metadata
I20260924 16:04:19.919667 3929 ts_tablet_manager.cc:1918] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:19.919850 3929 log.cc:1196] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/0a5bccaa61654268a52821feff8f53ca
I20260924 16:04:19.920395 3929 ts_tablet_manager.cc:1939] T 0a5bccaa61654268a52821feff8f53ca P 5526a7f538654b5792958c175cf3ccd0: Deleting consensus metadata
I20260924 16:04:19.921497 3402 catalog_manager.cc:4841] TS 537690d6912c4b40a3b0d389df3487ff (127.16.233.130:46489): tablet 0a5bccaa61654268a52821feff8f53ca (table test-table [id=01946c241a6040a08ca91d8059b4195f]) successfully deleted
I20260924 16:04:19.921976 3405 catalog_manager.cc:4841] TS 5526a7f538654b5792958c175cf3ccd0 (127.16.233.129:41373): tablet 0a5bccaa61654268a52821feff8f53ca (table test-table [id=01946c241a6040a08ca91d8059b4195f]) successfully deleted
Sep 24 16:04:19 dist-test-slave-xn4h krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 16:04:19.938382 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:19.905188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.16.233.129:41373)
0924 16:04:19.905444 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:19.905453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:19.918856 (+ 13403us) server_negotiation.cc:407] Connection header received
0924 16:04:19.919127 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:19.919132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:19.919251 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:19.919352 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:19.923646 (+ 4294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.924842 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.926179 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:19.926927 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:19.927073 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:19.932090 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:19.932106 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:19.932121 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:19.932165 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:19.935318 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:19.936102 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:19.936110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:19.936113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:19.936258 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:19.936667 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:19.936671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:19.936674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:19.937033 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:19.937404 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:19.937681 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 16:04:19.937940 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:04:19.939168 3606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:20.051234 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 3481
I20260924 16:04:20.067255 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 3618
Sep 24 16:04:20 dist-test-slave-xn4h krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:04:20.082129 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:20.068041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.16.233.130:46489)
0924 16:04:20.068313 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:20.068317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:20.068337 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:04:20.068451 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:20.068455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:20.068519 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:20.068606 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:20.069906 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:20.070735 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:20.071989 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:20.072366 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:20.072455 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:20.076709 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:20.076723 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:20.076740 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:20.076780 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:20.079704 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:20.080324 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:20.080330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:20.080333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:20.080447 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:20.080778 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:20.080781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:20.080783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:20.081158 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:20.081370 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:20.081600 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 16:04:20.081754 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:04:20.082778 3742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:20.174593 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 3618
I20260924 16:04:20.191627 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 3754
Sep 24 16:04:20 dist-test-slave-xn4h krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 16:04:20.206957 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:20.192384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.16.233.131:38773)
0924 16:04:20.192550 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:20.192554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:20.192573 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:04:20.192700 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:20.192704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:20.192780 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:20.192865 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:20.194138 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:20.194896 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:20.196292 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:20.196732 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:20.196872 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:20.201566 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:20.201581 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:20.201597 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:20.201635 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:20.204409 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:20.205055 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:20.205082 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:20.205087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:20.205241 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:20.205643 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:20.205646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:20.205648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:20.206002 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:20.206209 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:20.206539 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 16:04:20.206697 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:04:20.207599 3877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:20.295544 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 3754
I20260924 16:04:20.312474 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 3386
I20260924 16:04:20.313549 3449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:20.422396 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 3386
2026-09-24T16:04:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4334 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T16:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:20Z Disabled control of system clock
I20260924 16:04:20.502247 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:37765
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:37765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:04:20.840072 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:20.840360 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:20.840458 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:20.847824 3938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:20.847940 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:20.847980 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:20.848039 3938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:20.848084 3938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:20.857404 3938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:37765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:37765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:20.858687 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:20.860498 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:20.872357 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:20.872392 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:20.872432 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:20.873304 3938 server_base.cc:1034] running on GCE node
I20260924 16:04:20.874087 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:20.875684 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:20.876932 3938 hybrid_clock.cc:648] HybridClock initialized: now 1790265860876880 us; error 61 us; skew 500 ppm
I20260924 16:04:20.877311 3938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:20.886407 3938 webserver.cc:460] Webserver started at http://127.16.233.190:35269/ using document root <none> and password file <none>
I20260924 16:04:20.887184 3938 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:20.887331 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:20.887677 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:20.890829 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "2232f631ad204907b5bf7c3d81d47361"
format_stamp: "Formatted at 2026-09-24 16:04:20 on dist-test-slave-xn4h"
server_key: "5385e55544f8ce52df50aaa9f5dd462b"
server_key_iv: "99c42c54a0070e1320345ff7b46c4663"
server_key_version: "encryptionkey@0"
I20260924 16:04:20.891688 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "2232f631ad204907b5bf7c3d81d47361"
format_stamp: "Formatted at 2026-09-24 16:04:20 on dist-test-slave-xn4h"
server_key: "5385e55544f8ce52df50aaa9f5dd462b"
server_key_iv: "99c42c54a0070e1320345ff7b46c4663"
server_key_version: "encryptionkey@0"
I20260924 16:04:20.897080 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:04:20.900650 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:20.902134 3938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:04:20.902387 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "2232f631ad204907b5bf7c3d81d47361"
format_stamp: "Formatted at 2026-09-24 16:04:20 on dist-test-slave-xn4h"
server_key: "5385e55544f8ce52df50aaa9f5dd462b"
server_key_iv: "99c42c54a0070e1320345ff7b46c4663"
server_key_version: "encryptionkey@0"
I20260924 16:04:20.902635 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:20.935570 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:20.936477 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:20.936789 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:20.955502 3938 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:37765
I20260924 16:04:20.955533 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:37765 every 8 connection(s)
I20260924 16:04:20.956882 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:20.957319 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 3938
I20260924 16:04:20.957504 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:20.958093 17318 external_mini_cluster.cc:1428] Setting key 79afcf7f6ed2e478f57a8083dff76c01
I20260924 16:04:20.961459 4005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:20.971478 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:20.960143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.16.233.190:37765)
0924 16:04:20.961140 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:20.961150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:20.961191 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:04:20.962953 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:20.962975 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:20.963388 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:20.963765 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:20.964349 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:20.965658 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:20.967103 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:20.967740 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:20.967943 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:20.969152 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:20.969159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:20.969172 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:20.969749 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:20.969830 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:20.969852 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:20.970008 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:20.970238 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 16:04:20.970570 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":368,"threads_started":1}
I20260924 16:04:20.972435 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361: Bootstrap starting.
I20260924 16:04:20.975666 4005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:20.976641 4005 log.cc:824] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:20.979250 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361: No bootstrap required, opened a new log
I20260924 16:04:20.986869 4005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f631ad204907b5bf7c3d81d47361" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37765 } }
I20260924 16:04:20.987299 4005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:20.987412 4005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2232f631ad204907b5bf7c3d81d47361, State: Initialized, Role: FOLLOWER
I20260924 16:04:20.988029 4005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f631ad204907b5bf7c3d81d47361" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37765 } }
I20260924 16:04:20.988282 4005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:20.988435 4005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:20.988626 4005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:20.991016 4005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f631ad204907b5bf7c3d81d47361" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37765 } }
I20260924 16:04:20.991700 4005 leader_election.cc:302] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2232f631ad204907b5bf7c3d81d47361; no voters:
I20260924 16:04:20.992357 4005 leader_election.cc:288] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:20.992470 4010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:20.993364 4010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [term 1 LEADER]: Becoming Leader. State: Replica: 2232f631ad204907b5bf7c3d81d47361, State: Running, Role: LEADER
I20260924 16:04:20.993906 4010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f631ad204907b5bf7c3d81d47361" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37765 } }
I20260924 16:04:20.994794 4005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:20.996878 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2232f631ad204907b5bf7c3d81d47361. Latest consensus state: current_term: 1 leader_uuid: "2232f631ad204907b5bf7c3d81d47361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f631ad204907b5bf7c3d81d47361" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37765 } } }
I20260924 16:04:20.997388 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:20.997505 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2232f631ad204907b5bf7c3d81d47361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f631ad204907b5bf7c3d81d47361" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 37765 } } }
I20260924 16:04:20.997807 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:20.998692 4018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:21.004281 4018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:21.012832 4018 catalog_manager.cc:1347] Generated new cluster ID: 24846da6e97840a2abc308d36dfe7a3f
I20260924 16:04:21.012954 4018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:21.031288 4018 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:21.032529 4018 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:21.045747 4018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361: Generated new TSK 0
I20260924 16:04:21.046653 4018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:21.052814 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37765
--builtin_ntp_servers=127.16.233.148:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:04:21.423112 4030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:21.423381 4030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:21.423554 4030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:21.430977 4030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:21.431109 4030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:21.431298 4030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:04:21.440651 4030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:21.441944 4030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:21.443768 4030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:21.456993 4035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:21.457068 4036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:21.457044 4038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:21.457894 4030 server_base.cc:1034] running on GCE node
I20260924 16:04:21.458566 4030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:21.459463 4030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:21.460662 4030 hybrid_clock.cc:648] HybridClock initialized: now 1790265861460615 us; error 45 us; skew 500 ppm
I20260924 16:04:21.461004 4030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:21.463295 4030 webserver.cc:460] Webserver started at http://127.16.233.129:40741/ using document root <none> and password file <none>
I20260924 16:04:21.463992 4030 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:21.464107 4030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:21.464439 4030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:21.467355 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40"
format_stamp: "Formatted at 2026-09-24 16:04:21 on dist-test-slave-xn4h"
server_key: "d3212dcaa79b04f8e77b8019e3a257f3"
server_key_iv: "bd86700ed1772ce6d0db6d99c520432c"
server_key_version: "encryptionkey@0"
I20260924 16:04:21.468137 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40"
format_stamp: "Formatted at 2026-09-24 16:04:21 on dist-test-slave-xn4h"
server_key: "d3212dcaa79b04f8e77b8019e3a257f3"
server_key_iv: "bd86700ed1772ce6d0db6d99c520432c"
server_key_version: "encryptionkey@0"
I20260924 16:04:21.473192 4030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:04:21.476696 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:21.478024 4030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:04:21.478215 4030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40"
format_stamp: "Formatted at 2026-09-24 16:04:21 on dist-test-slave-xn4h"
server_key: "d3212dcaa79b04f8e77b8019e3a257f3"
server_key_iv: "bd86700ed1772ce6d0db6d99c520432c"
server_key_version: "encryptionkey@0"
I20260924 16:04:21.478498 4030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:21.498999 4030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:21.499858 4030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:21.500176 4030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:21.501165 4030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:21.502764 4030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:21.502897 4030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:21.502990 4030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:21.503046 4030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:21.536679 4030 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:46749
I20260924 16:04:21.536772 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:46749 every 8 connection(s)
I20260924 16:04:21.538048 4030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:21.541765 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 4030
I20260924 16:04:21.541956 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:04:21.542488 17318 external_mini_cluster.cc:1428] Setting key f90b07e08db12ed2cd51aa33c9887dd9
I20260924 16:04:21.545750 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37765
--builtin_ntp_servers=127.16.233.148:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:04:21.552691 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:21.540724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:41853 (local address 127.16.233.190:37765)
0924 16:04:21.541212 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:21.541217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:21.543085 (+ 1868us) server_negotiation.cc:407] Connection header received
0924 16:04:21.545095 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:21.545100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:21.545179 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:21.545321 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:21.546595 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:21.547810 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:21.549399 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:21.550142 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:21.550285 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:21.551692 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:21.551702 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:21.551710 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:21.551737 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:21.551794 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:21.551813 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:21.551905 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:21.552125 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 16:04:21.552242 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":229,"threads_started":1}
I20260924 16:04:21.553498 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:21.541328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:37765 (local address 127.16.233.129:41853)
0924 16:04:21.542404 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:21.542471 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:21.544784 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:21.545617 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:21.545626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:21.545722 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:21.546429 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:21.546449 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:21.548008 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:21.548012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:21.549246 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:21.549258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:21.550400 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:21.550406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:21.550622 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:21.551346 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:04:21.551349 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:21.551390 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:04:21.551489 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:21.551908 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:21.551918 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:21.551924 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:04:21.552061 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 16:04:21.552372 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":346,"threads_started":1}
I20260924 16:04:21.555176 4161 heartbeater.cc:347] Connected to a master server at 127.16.233.190:37765
I20260924 16:04:21.555537 4161 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:21.556298 4161 heartbeater.cc:511] Master 127.16.233.190:37765 requested a full tablet report, sending...
I20260924 16:04:21.558645 3969 ts_manager.cc:194] Registered new tserver with Master: 80d86e8ebc3b4dd9bd68533d7aaf8d40 (127.16.233.129:46749)
I20260924 16:04:21.560935 3969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.129:41853
W20260924 16:04:21.877338 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:21.877627 4165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:21.877718 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:21.885113 4165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:21.885231 4165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:21.885342 4165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:04:21.894642 4165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:21.896198 4165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:21.897919 4165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:21.911645 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:21.911566 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:21.911605 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:21.912407 4165 server_base.cc:1034] running on GCE node
I20260924 16:04:21.913033 4165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:21.914021 4165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:21.915246 4165 hybrid_clock.cc:648] HybridClock initialized: now 1790265861915172 us; error 71 us; skew 500 ppm
I20260924 16:04:21.915588 4165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:21.917752 4165 webserver.cc:460] Webserver started at http://127.16.233.130:41949/ using document root <none> and password file <none>
I20260924 16:04:21.918498 4165 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:21.918629 4165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:21.918943 4165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:21.921764 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "a2c300831784485d82548ccfc88d12a7"
format_stamp: "Formatted at 2026-09-24 16:04:21 on dist-test-slave-xn4h"
server_key: "db40652749f05f3d956e38ca64defbbe"
server_key_iv: "ca5bdf6ea6213963e832cacff9e3d417"
server_key_version: "encryptionkey@0"
I20260924 16:04:21.922612 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "a2c300831784485d82548ccfc88d12a7"
format_stamp: "Formatted at 2026-09-24 16:04:21 on dist-test-slave-xn4h"
server_key: "db40652749f05f3d956e38ca64defbbe"
server_key_iv: "ca5bdf6ea6213963e832cacff9e3d417"
server_key_version: "encryptionkey@0"
I20260924 16:04:21.927711 4165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 16:04:21.931222 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:21.932539 4165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:04:21.932713 4165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "a2c300831784485d82548ccfc88d12a7"
format_stamp: "Formatted at 2026-09-24 16:04:21 on dist-test-slave-xn4h"
server_key: "db40652749f05f3d956e38ca64defbbe"
server_key_iv: "ca5bdf6ea6213963e832cacff9e3d417"
server_key_version: "encryptionkey@0"
I20260924 16:04:21.932926 4165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:21.958993 4165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:21.959869 4165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:21.960197 4165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:21.961247 4165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:21.962939 4165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:21.963056 4165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:21.963191 4165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:21.963285 4165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:21.997450 4165 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:45233
I20260924 16:04:21.997545 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:45233 every 8 connection(s)
I20260924 16:04:21.998812 4165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:22.004588 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 4165
I20260924 16:04:22.004804 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:04:22.005347 17318 external_mini_cluster.cc:1428] Setting key f16a4f0d63da7517bf4412e04ef4d194
I20260924 16:04:22.008338 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37765
--builtin_ntp_servers=127.16.233.148:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:04:22.012085 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.001373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:48397 (local address 127.16.233.190:37765)
0924 16:04:22.001599 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:22.001605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:22.003007 (+ 1402us) server_negotiation.cc:407] Connection header received
0924 16:04:22.004868 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:22.004877 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:22.004984 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:22.005093 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:22.006297 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.007223 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.008979 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.009729 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.009876 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.011196 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:22.011204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:22.011212 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:22.011236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:22.011279 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:22.011296 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:22.011389 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:22.011585 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 16:04:22.011694 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 16:04:22.012892 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.001939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:37765 (local address 127.16.233.130:48397)
0924 16:04:22.002852 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:22.002886 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:22.004563 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:22.005295 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:22.005304 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:22.005426 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:22.006142 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.006163 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.007371 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.007378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.008833 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.008845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.009932 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.009937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.010132 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.010882 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:04:22.010884 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:22.010929 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:04:22.011010 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:22.011390 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:22.011397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:22.011404 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:04:22.011540 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 16:04:22.011820 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":305,"threads_started":1}
I20260924 16:04:22.014359 4296 heartbeater.cc:347] Connected to a master server at 127.16.233.190:37765
I20260924 16:04:22.014658 4296 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:22.015368 4296 heartbeater.cc:511] Master 127.16.233.190:37765 requested a full tablet report, sending...
I20260924 16:04:22.016775 3969 ts_manager.cc:194] Registered new tserver with Master: a2c300831784485d82548ccfc88d12a7 (127.16.233.130:45233)
I20260924 16:04:22.017581 3969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.130:48397
W20260924 16:04:22.340925 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:22.341270 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:22.341449 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:22.348681 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:22.348825 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:22.349013 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:04:22.358191 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:37765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:22.359539 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:22.361325 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:22.374190 4304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:22.374305 4305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:22.374244 4307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:22.379700 4299 server_base.cc:1034] running on GCE node
I20260924 16:04:22.380342 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:22.381331 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:22.382560 4299 hybrid_clock.cc:648] HybridClock initialized: now 1790265862382530 us; error 54 us; skew 500 ppm
I20260924 16:04:22.382881 4299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:22.385044 4299 webserver.cc:460] Webserver started at http://127.16.233.131:44311/ using document root <none> and password file <none>
I20260924 16:04:22.385738 4299 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:22.385850 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:22.386176 4299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:22.389074 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "cedc99deb7404f4fa52b20ad1af88061"
format_stamp: "Formatted at 2026-09-24 16:04:22 on dist-test-slave-xn4h"
server_key: "4afe118fbcabd7f9e26805a31092de33"
server_key_iv: "1fe3e5dba98c348d4d6f4e37f01e30d8"
server_key_version: "encryptionkey@0"
I20260924 16:04:22.389845 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "cedc99deb7404f4fa52b20ad1af88061"
format_stamp: "Formatted at 2026-09-24 16:04:22 on dist-test-slave-xn4h"
server_key: "4afe118fbcabd7f9e26805a31092de33"
server_key_iv: "1fe3e5dba98c348d4d6f4e37f01e30d8"
server_key_version: "encryptionkey@0"
I20260924 16:04:22.394891 4299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:04:22.398353 4313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:22.399637 4299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:04:22.399792 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "cedc99deb7404f4fa52b20ad1af88061"
format_stamp: "Formatted at 2026-09-24 16:04:22 on dist-test-slave-xn4h"
server_key: "4afe118fbcabd7f9e26805a31092de33"
server_key_iv: "1fe3e5dba98c348d4d6f4e37f01e30d8"
server_key_version: "encryptionkey@0"
I20260924 16:04:22.400015 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:22.417141 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:22.417949 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:22.418251 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:22.419363 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:22.420763 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:22.420882 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:22.421028 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:22.421135 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:22.453923 4299 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:38049
I20260924 16:04:22.454000 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:38049 every 8 connection(s)
I20260924 16:04:22.455294 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:22.463701 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 4299
I20260924 16:04:22.464005 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:04:22.464452 17318 external_mini_cluster.cc:1428] Setting key 60d43ba59681fdd3c8422f893ab8f419
I20260924 16:04:22.468225 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.457854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:42221 (local address 127.16.233.190:37765)
0924 16:04:22.458054 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:22.458059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:22.459425 (+ 1366us) server_negotiation.cc:407] Connection header received
0924 16:04:22.461235 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:22.461240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:22.461331 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:22.461441 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:22.462535 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.463289 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.465089 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.465620 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.465761 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.467368 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:22.467372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:22.467376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:22.467392 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:22.467425 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:22.467437 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:22.467538 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:22.467763 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 16:04:22.467862 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:04:22.469134 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.458421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:37765 (local address 127.16.233.131:42221)
0924 16:04:22.459278 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:22.459317 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:22.460939 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:22.461670 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:22.461677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:22.461758 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:22.462373 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.462392 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.463461 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.463473 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.464914 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.464929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.465826 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.465831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.466119 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.466888 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:04:22.466891 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:22.466929 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:04:22.467015 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:22.467536 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:22.467549 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:22.467560 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:04:22.467734 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 16:04:22.468125 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":258,"threads_started":1}
I20260924 16:04:22.470404 4429 heartbeater.cc:347] Connected to a master server at 127.16.233.190:37765
I20260924 16:04:22.470692 4429 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:22.471444 4429 heartbeater.cc:511] Master 127.16.233.190:37765 requested a full tablet report, sending...
I20260924 16:04:22.472786 3969 ts_manager.cc:194] Registered new tserver with Master: cedc99deb7404f4fa52b20ad1af88061 (127.16.233.131:38049)
I20260924 16:04:22.473554 3969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.131:42221
I20260924 16:04:22.479516 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:22.489638 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.483593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.16.233.190:37765)
0924 16:04:22.483752 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:22.483775 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:22.484230 (+ 455us) server_negotiation.cc:407] Connection header received
0924 16:04:22.484454 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:22.484458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:22.484517 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:22.484614 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:22.485116 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.485825 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.487216 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.487621 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.487755 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.488892 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:22.488897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:22.488900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:22.488919 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:22.488954 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:22.488967 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:22.489120 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:22.489315 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 16:04:22.489371 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:04:22.493350 3969 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:22.500424 3969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:22.532805 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.512380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:38049 (local address 127.0.0.1:34922)
0924 16:04:22.521669 (+ 9289us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:22.521714 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:22.521884 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:22.523838 (+ 1954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:22.523847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:22.523880 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:22.524482 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.524498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.527233 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.527241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.530142 (+ 2901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.530157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.531849 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.531855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.532048 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.532102 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 16:04:22.532232 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 16:04:22.532411 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9037,"mutex_wait_us":15,"thread_start_us":9633,"threads_started":1}
I20260924 16:04:22.532824 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.512582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.16.233.131:38049)
0924 16:04:22.523151 (+ 10569us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:22.523161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:22.523203 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:04:22.523304 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:22.523312 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:22.523592 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:22.524715 (+ 1123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:22.524989 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.527023 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.530353 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.531661 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.531841 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.532049 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:22.532278 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 16:04:22.532433 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10384,"thread_start_us":10246,"threads_started":1}
I20260924 16:04:22.534135 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.512750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46749 (local address 127.0.0.1:46944)
0924 16:04:22.522376 (+ 9626us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:22.522398 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:22.522542 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:22.524023 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:22.524028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:22.524052 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:22.524623 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.524645 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.527176 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.527185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.531167 (+ 3982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.531177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.533283 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.533293 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.533522 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.533555 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:04:22.533632 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:04:22.533750 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9421,"mutex_wait_us":1811,"thread_start_us":8489,"threads_started":1}
I20260924 16:04:22.535128 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.513064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.16.233.129:46749)
0924 16:04:22.523446 (+ 10382us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:22.523454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:22.523481 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:04:22.523572 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:22.523579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:22.523796 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:22.524266 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:22.524897 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.526982 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.531345 (+ 4363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.532552 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.532710 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.534577 (+ 1867us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:22.534645 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 16:04:22.534816 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10224,"thread_start_us":10118,"threads_started":1}
I20260924 16:04:22.537501 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.525462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:45233 (local address 127.0.0.1:54426)
0924 16:04:22.529778 (+ 4316us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:22.529794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:22.529937 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:22.530532 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:22.530537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:22.530562 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:22.530932 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.530943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.532600 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.532605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.534772 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.534786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.536793 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.536799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.536954 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.536986 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:04:22.537069 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 16:04:22.537218 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4187,"thread_start_us":4051,"threads_started":1}
I20260924 16:04:22.537202 4363 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3ebe8f01a541fca9bb1e37173c3a18 (DEFAULT_TABLE table=test-table [id=ff922742b0c2430ab857f0c1f5ed0922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:22.538221 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.522784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.16.233.130:45233)
0924 16:04:22.528728 (+ 5944us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:22.528736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:22.529893 (+ 1157us) server_negotiation.cc:407] Connection header received
0924 16:04:22.530102 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:22.530108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:22.530311 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:22.530521 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:22.531089 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.532438 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.534966 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.536300 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.536487 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.536667 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:22.537703 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0924 16:04:22.537869 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5780,"thread_start_us":5673,"threads_started":1}
I20260924 16:04:22.538802 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3ebe8f01a541fca9bb1e37173c3a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:22.538455 4094 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3ebe8f01a541fca9bb1e37173c3a18 (DEFAULT_TABLE table=test-table [id=ff922742b0c2430ab857f0c1f5ed0922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:22.541818 4230 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3ebe8f01a541fca9bb1e37173c3a18 (DEFAULT_TABLE table=test-table [id=ff922742b0c2430ab857f0c1f5ed0922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:22.542498 4094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3ebe8f01a541fca9bb1e37173c3a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:22.543118 4230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3ebe8f01a541fca9bb1e37173c3a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:22.553210 4447 tablet_bootstrap.cc:492] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: Bootstrap starting.
I20260924 16:04:22.557529 4447 tablet_bootstrap.cc:654] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:22.557802 4448 tablet_bootstrap.cc:492] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: Bootstrap starting.
I20260924 16:04:22.558925 4447 log.cc:824] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:22.560375 4449 tablet_bootstrap.cc:492] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40: Bootstrap starting.
I20260924 16:04:22.561862 4448 tablet_bootstrap.cc:654] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:22.562635 4447 tablet_bootstrap.cc:492] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: No bootstrap required, opened a new log
I20260924 16:04:22.562920 4447 ts_tablet_manager.cc:1397] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 16:04:22.563191 4448 log.cc:824] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:22.563498 4449 tablet_bootstrap.cc:654] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:22.564208 4161 heartbeater.cc:503] Master 127.16.233.190:37765 was elected leader, sending a full tablet report...
I20260924 16:04:22.564843 4449 log.cc:824] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:22.566538 4448 tablet_bootstrap.cc:492] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: No bootstrap required, opened a new log
I20260924 16:04:22.566944 4448 ts_tablet_manager.cc:1397] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260924 16:04:22.572827 4447 raft_consensus.cc:348] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } }
I20260924 16:04:22.573093 4449 tablet_bootstrap.cc:492] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40: No bootstrap required, opened a new log
I20260924 16:04:22.573230 4447 raft_consensus.cc:374] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:22.573336 4447 raft_consensus.cc:729] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedc99deb7404f4fa52b20ad1af88061, State: Initialized, Role: FOLLOWER
I20260924 16:04:22.573366 4449 ts_tablet_manager.cc:1397] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260924 16:04:22.573944 4447 consensus_queue.cc:260] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } }
I20260924 16:04:22.575074 4429 heartbeater.cc:503] Master 127.16.233.190:37765 was elected leader, sending a full tablet report...
I20260924 16:04:22.575918 4447 ts_tablet_manager.cc:1428] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260924 16:04:22.578891 4448 raft_consensus.cc:348] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } }
I20260924 16:04:22.579339 4448 raft_consensus.cc:374] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:22.579474 4448 raft_consensus.cc:729] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c300831784485d82548ccfc88d12a7, State: Initialized, Role: FOLLOWER
I20260924 16:04:22.580147 4448 consensus_queue.cc:260] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } }
I20260924 16:04:22.581275 4296 heartbeater.cc:503] Master 127.16.233.190:37765 was elected leader, sending a full tablet report...
I20260924 16:04:22.581955 4448 ts_tablet_manager.cc:1428] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260924 16:04:22.584033 4449 raft_consensus.cc:348] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } }
I20260924 16:04:22.584398 4449 raft_consensus.cc:374] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:22.584509 4449 raft_consensus.cc:729] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d86e8ebc3b4dd9bd68533d7aaf8d40, State: Initialized, Role: FOLLOWER
I20260924 16:04:22.585124 4449 consensus_queue.cc:260] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } }
I20260924 16:04:22.586815 4449 ts_tablet_manager.cc:1428] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260924 16:04:22.707091 4430 tablet.cc:2369] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:04:22.751011 4297 tablet.cc:2369] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:04:22.790571 4162 tablet.cc:2369] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:04:22.907433 4455 raft_consensus.cc:482] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:22.907800 4455 raft_consensus.cc:504] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } }
I20260924 16:04:22.909512 4455 leader_election.cc:288] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cedc99deb7404f4fa52b20ad1af88061 (127.16.233.131:38049), a2c300831784485d82548ccfc88d12a7 (127.16.233.130:45233)
I20260924 16:04:22.915922 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.909948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:47057 (local address 127.16.233.131:38049)
0924 16:04:22.910128 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:22.910132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:22.910619 (+ 487us) server_negotiation.cc:407] Connection header received
0924 16:04:22.910895 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:22.910904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:22.911004 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:22.911066 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:22.911801 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.912516 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.914236 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.915049 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.915218 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.915288 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:22.915563 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 16:04:22.915686 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:04:22.916015 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.909816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:38049 (local address 127.16.233.129:47057)
0924 16:04:22.910508 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:22.910535 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:22.910679 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:22.911267 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:22.911276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:22.911297 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:22.911640 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.911649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.912643 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.912651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.914029 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.914042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.915234 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.915241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.915402 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.915426 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:04:22.915534 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:04:22.915661 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"spinlock_wait_cycles":18304,"thread_start_us":344,"threads_started":1}
I20260924 16:04:22.916379 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.910873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:40501 (local address 127.16.233.130:45233)
0924 16:04:22.911084 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:22.911090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:22.911390 (+ 300us) server_negotiation.cc:407] Connection header received
0924 16:04:22.911611 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:22.911622 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:22.911690 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:22.911786 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:22.912304 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.912976 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.914756 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.915480 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.915674 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.915741 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:22.916016 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 16:04:22.916138 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:04:22.916419 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.910665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:45233 (local address 127.16.233.129:40501)
0924 16:04:22.911254 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:22.911273 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:22.911471 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:22.911800 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:22.911804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:22.911826 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:22.912155 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.912163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.913106 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.913111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.914583 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:22.914595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.915675 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.915681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:22.915843 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.915866 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:04:22.915989 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 16:04:22.916117 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":305,"threads_started":1}
I20260924 16:04:22.917014 4384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3ebe8f01a541fca9bb1e37173c3a18" candidate_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedc99deb7404f4fa52b20ad1af88061" is_pre_election: true
I20260924 16:04:22.917533 4384 raft_consensus.cc:2454] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80d86e8ebc3b4dd9bd68533d7aaf8d40 in term 0.
I20260924 16:04:22.918188 4047 leader_election.cc:302] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80d86e8ebc3b4dd9bd68533d7aaf8d40, cedc99deb7404f4fa52b20ad1af88061; no voters:
I20260924 16:04:22.918169 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3ebe8f01a541fca9bb1e37173c3a18" candidate_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c300831784485d82548ccfc88d12a7" is_pre_election: true
I20260924 16:04:22.918720 4455 raft_consensus.cc:2781] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:22.918731 4251 raft_consensus.cc:2454] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80d86e8ebc3b4dd9bd68533d7aaf8d40 in term 0.
I20260924 16:04:22.918883 4455 raft_consensus.cc:482] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:22.918980 4455 raft_consensus.cc:3037] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:22.921306 4455 raft_consensus.cc:504] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } }
I20260924 16:04:22.921984 4455 leader_election.cc:288] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [CANDIDATE]: Term 1 election: Requested vote from peers cedc99deb7404f4fa52b20ad1af88061 (127.16.233.131:38049), a2c300831784485d82548ccfc88d12a7 (127.16.233.130:45233)
I20260924 16:04:22.922401 4384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3ebe8f01a541fca9bb1e37173c3a18" candidate_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedc99deb7404f4fa52b20ad1af88061"
I20260924 16:04:22.922503 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3ebe8f01a541fca9bb1e37173c3a18" candidate_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c300831784485d82548ccfc88d12a7"
I20260924 16:04:22.922603 4384 raft_consensus.cc:3037] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:22.922736 4251 raft_consensus.cc:3037] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:22.924803 4384 raft_consensus.cc:2454] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d86e8ebc3b4dd9bd68533d7aaf8d40 in term 1.
I20260924 16:04:22.924949 4251 raft_consensus.cc:2454] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d86e8ebc3b4dd9bd68533d7aaf8d40 in term 1.
I20260924 16:04:22.925240 4047 leader_election.cc:302] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80d86e8ebc3b4dd9bd68533d7aaf8d40, cedc99deb7404f4fa52b20ad1af88061; no voters:
I20260924 16:04:22.925537 4455 raft_consensus.cc:2781] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:22.925963 4455 raft_consensus.cc:686] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 1 LEADER]: Becoming Leader. State: Replica: 80d86e8ebc3b4dd9bd68533d7aaf8d40, State: Running, Role: LEADER
I20260924 16:04:22.926501 4455 consensus_queue.cc:237] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } }
I20260924 16:04:22.931128 3968 catalog_manager.cc:5495] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80d86e8ebc3b4dd9bd68533d7aaf8d40 (127.16.233.129). New cstate: current_term: 1 leader_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d86e8ebc3b4dd9bd68533d7aaf8d40" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 46749 } health_report { overall_health: HEALTHY } } }
I20260924 16:04:22.987196 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:22.981568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.16.233.129:46749)
0924 16:04:22.981781 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:22.981789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:22.981809 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:04:22.982064 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:22.982068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:22.982159 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:22.982360 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:22.982766 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.983450 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.984837 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:22.985277 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:22.985419 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:22.986014 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:22.986580 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:22.986765 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:22.986883 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 16:04:22.986926 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:04:22.995640 4384 raft_consensus.cc:1261] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [term 1 FOLLOWER]: Refusing update from remote peer 80d86e8ebc3b4dd9bd68533d7aaf8d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:22.995710 4251 raft_consensus.cc:1261] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [term 1 FOLLOWER]: Refusing update from remote peer 80d86e8ebc3b4dd9bd68533d7aaf8d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:22.996727 4458 consensus_queue.cc:1035] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cedc99deb7404f4fa52b20ad1af88061" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:22.997359 4455 consensus_queue.cc:1035] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c300831784485d82548ccfc88d12a7" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:23.037074 3968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:23.037441 3968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:23.041286 3968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2232f631ad204907b5bf7c3d81d47361: Sending DeleteTablet for 3 replicas of tablet 4d3ebe8f01a541fca9bb1e37173c3a18
I20260924 16:04:23.042580 4094 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3ebe8f01a541fca9bb1e37173c3a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:23 UTC) from {username='slave'} at 127.0.0.1:46944
I20260924 16:04:23.042583 4363 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3ebe8f01a541fca9bb1e37173c3a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:23 UTC) from {username='slave'} at 127.0.0.1:34922
I20260924 16:04:23.042601 4230 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3ebe8f01a541fca9bb1e37173c3a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:23 UTC) from {username='slave'} at 127.0.0.1:54426
I20260924 16:04:23.043576 4470 tablet_replica.cc:330] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: stopping tablet replica
I20260924 16:04:23.044051 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 4030
I20260924 16:04:23.044201 4470 raft_consensus.cc:2229] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:23.044802 4470 raft_consensus.cc:2258] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:23.047497 4470 ts_tablet_manager.cc:1905] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:23.055569 4470 ts_tablet_manager.cc:1918] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:23.055871 4470 log.cc:1196] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/4d3ebe8f01a541fca9bb1e37173c3a18
I20260924 16:04:23.056298 4472 tablet_replica.cc:330] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40: stopping tablet replica
I20260924 16:04:23.056530 4470 ts_tablet_manager.cc:1939] T 4d3ebe8f01a541fca9bb1e37173c3a18 P cedc99deb7404f4fa52b20ad1af88061: Deleting consensus metadata
I20260924 16:04:23.057075 4472 raft_consensus.cc:2229] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:04:23.057544 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.045959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.16.233.129:46749)
0924 16:04:23.046183 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:23.046193 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:23.046549 (+ 356us) server_negotiation.cc:407] Connection header received
0924 16:04:23.046832 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:23.046838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:23.046958 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:23.047058 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:23.047939 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:23.049527 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:23.052135 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:23.053125 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:23.053253 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:23.055161 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:23.055177 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:23.055205 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:23.056249 (+ 1044us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:23.056322 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:23.056353 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:23.056437 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:23.057015 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 16:04:23.057153 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:04:23.058091 4472 raft_consensus.cc:2258] T 4d3ebe8f01a541fca9bb1e37173c3a18 P 80d86e8ebc3b4dd9bd68533d7aaf8d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:23.058336 4156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:23.058408 4471 tablet_replica.cc:330] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: stopping tablet replica
I20260924 16:04:23.058651 3955 catalog_manager.cc:4841] TS cedc99deb7404f4fa52b20ad1af88061 (127.16.233.131:38049): tablet 4d3ebe8f01a541fca9bb1e37173c3a18 (table test-table [id=ff922742b0c2430ab857f0c1f5ed0922]) successfully deleted
I20260924 16:04:23.058887 4471 raft_consensus.cc:2229] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:23.059376 4471 raft_consensus.cc:2258] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:23.061398 4471 ts_tablet_manager.cc:1905] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:23.067130 4471 ts_tablet_manager.cc:1918] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:23.067319 4471 log.cc:1196] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/4d3ebe8f01a541fca9bb1e37173c3a18
I20260924 16:04:23.067782 4471 ts_tablet_manager.cc:1939] T 4d3ebe8f01a541fca9bb1e37173c3a18 P a2c300831784485d82548ccfc88d12a7: Deleting consensus metadata
I20260924 16:04:23.069393 3955 catalog_manager.cc:4841] TS a2c300831784485d82548ccfc88d12a7 (127.16.233.130:45233): tablet 4d3ebe8f01a541fca9bb1e37173c3a18 (table test-table [id=ff922742b0c2430ab857f0c1f5ed0922]) successfully deleted
I20260924 16:04:23.155038 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 4030
I20260924 16:04:23.173620 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 4165
W20260924 16:04:23.173673 3956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.129:46749 (error 108)
I20260924 16:04:23.175748 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.174897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46749 (local address 127.0.0.1:46970)
0924 16:04:23.175393 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:23.175521 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":216,"threads_started":1}
W20260924 16:04:23.176023 3956 catalog_manager.cc:4584] TS 80d86e8ebc3b4dd9bd68533d7aaf8d40 (127.16.233.129:46749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d3ebe8f01a541fca9bb1e37173c3a18: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46749: connect: Connection refused (error 111)
I20260924 16:04:23.180949 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.174455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.16.233.130:45233)
0924 16:04:23.174643 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:23.174649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:23.174740 (+ 91us) server_negotiation.cc:407] Connection header received
0924 16:04:23.174967 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:23.174973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:23.175120 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:23.175228 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:23.175699 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:23.176643 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:23.177900 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:23.178287 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:23.178586 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:23.179620 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:23.179628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:23.179641 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:23.180238 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:23.180282 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:23.180301 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:23.180387 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:23.180504 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 16:04:23.180589 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:04:23.181594 4291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:23.229852 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.229268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46749 (local address 127.0.0.1:46972)
0924 16:04:23.229560 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:23.229672 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 16:04:23.230963 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.230557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46749 (local address 127.0.0.1:46986)
0924 16:04:23.230726 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:23.230808 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:04:23.270624 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 4165
I20260924 16:04:23.288570 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 4299
I20260924 16:04:23.291317 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.290879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46749 (local address 127.0.0.1:46990)
0924 16:04:23.291064 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:23.291145 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 16:04:23.292495 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.292064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46749 (local address 127.0.0.1:46992)
0924 16:04:23.292231 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:23.292319 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:04:23.296061 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.289386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.16.233.131:38049)
0924 16:04:23.289557 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:23.289561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:23.289623 (+ 62us) server_negotiation.cc:407] Connection header received
0924 16:04:23.289799 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:23.289808 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:23.289897 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:23.289988 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:23.290598 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:23.291501 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:23.293150 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:23.293535 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:23.293629 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:23.294711 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:23.294720 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:23.294734 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:23.295343 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:23.295390 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:23.295436 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:23.295537 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:23.295631 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 16:04:23.295713 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:04:23.296720 4424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:23.373128 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.372615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46749 (local address 127.0.0.1:47000)
0924 16:04:23.372876 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:23.372983 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 16:04:23.374243 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:23.373814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:46749 (local address 127.0.0.1:47012)
0924 16:04:23.374042 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:23.374112 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 16:04:23.384915 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 4299
I20260924 16:04:23.401870 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 3938
I20260924 16:04:23.403112 4000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:23.513449 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 3938
2026-09-24T16:04:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3089 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:23 dist-test-slave-xn4h krb5kdc[4482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:23 dist-test-slave-xn4h krb5kdc[4482](info): set up 2 sockets
Sep 24 16:04:23 dist-test-slave-xn4h krb5kdc[4482](info): commencing operation
krb5kdc: starting...
W20260924 16:04:24.636101 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:24 dist-test-slave-xn4h krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:04:24.899194 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41931
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:41931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:25.236334 4497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:25.236616 4497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:25.236779 4497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:25.243927 4497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:25.244025 4497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:25.244071 4497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:25.244156 4497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:25.244237 4497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:25.253813 4497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:41931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:41931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:25.255152 4497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:25.256497 4497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:25.268159 4502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:25.268195 4505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:25.268105 4503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:25.268987 4497 server_base.cc:1034] running on GCE node
I20260924 16:04:25.269806 4497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:25.271260 4497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:25.272500 4497 hybrid_clock.cc:648] HybridClock initialized: now 1790265865272448 us; error 52 us; skew 500 ppm
I20260924 16:04:25.272864 4497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:25 dist-test-slave-xn4h krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265865, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:25.275939 4497 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:04:25.277325 4497 webserver.cc:460] Webserver started at http://127.16.233.190:44119/ using document root <none> and password file <none>
I20260924 16:04:25.277930 4497 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:25.278026 4497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:25.278265 4497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:25.281148 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "f86e8e7ee7f348269c2da9a7d18daed5"
format_stamp: "Formatted at 2026-09-24 16:04:25 on dist-test-slave-xn4h"
server_key: "c63ba376b90e7fa283ea2491583d4f4c"
server_key_iv: "4045aed6b0041ef5afebe1c8ae8dd156"
server_key_version: "encryptionkey@0"
I20260924 16:04:25.281836 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "f86e8e7ee7f348269c2da9a7d18daed5"
format_stamp: "Formatted at 2026-09-24 16:04:25 on dist-test-slave-xn4h"
server_key: "c63ba376b90e7fa283ea2491583d4f4c"
server_key_iv: "4045aed6b0041ef5afebe1c8ae8dd156"
server_key_version: "encryptionkey@0"
I20260924 16:04:25.286695 4497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 16:04:25.289803 4512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:25.291072 4497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:25.291236 4497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "f86e8e7ee7f348269c2da9a7d18daed5"
format_stamp: "Formatted at 2026-09-24 16:04:25 on dist-test-slave-xn4h"
server_key: "c63ba376b90e7fa283ea2491583d4f4c"
server_key_iv: "4045aed6b0041ef5afebe1c8ae8dd156"
server_key_version: "encryptionkey@0"
I20260924 16:04:25.291395 4497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:25.327097 4497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:25.331259 4497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:25.331513 4497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:25.350063 4497 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:41931
I20260924 16:04:25.350085 4564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:41931 every 8 connection(s)
I20260924 16:04:25.351521 4497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:25.354395 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 4497
I20260924 16:04:25.354588 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:25.355108 17318 external_mini_cluster.cc:1428] Setting key ec11895c93245588a9c00ebb72176566
I20260924 16:04:25.355566 4565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:25.370662 4565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5: Bootstrap starting.
Sep 24 16:04:25 dist-test-slave-xn4h krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:25.373925 4565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:25.375093 4565 log.cc:824] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:25.378512 4565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5: No bootstrap required, opened a new log
I20260924 16:04:25.379529 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:25.356986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.16.233.190:41931)
0924 16:04:25.357793 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:25.357804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:25.357843 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:04:25.360173 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:25.360195 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:25.360608 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:25.360948 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:25.362882 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:25.364182 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:25.366091 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:25.366894 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:25.367034 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:25.371894 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:25.371902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:25.371917 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:25.371962 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:25.375182 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:25.376110 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:25.376121 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:25.376134 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:25.376367 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:25.376796 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:25.376799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:25.376801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:25.377360 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:25.377575 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:25.378042 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 16:04:25.378445 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":350,"threads_started":1}
I20260924 16:04:25.385910 4565 raft_consensus.cc:348] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e8e7ee7f348269c2da9a7d18daed5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41931 } }
I20260924 16:04:25.386328 4565 raft_consensus.cc:374] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:25.386444 4565 raft_consensus.cc:729] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86e8e7ee7f348269c2da9a7d18daed5, State: Initialized, Role: FOLLOWER
I20260924 16:04:25.387044 4565 consensus_queue.cc:260] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e8e7ee7f348269c2da9a7d18daed5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41931 } }
I20260924 16:04:25.387341 4565 raft_consensus.cc:388] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:25.387501 4565 raft_consensus.cc:482] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:25.387669 4565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:25.389843 4565 raft_consensus.cc:504] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e8e7ee7f348269c2da9a7d18daed5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41931 } }
I20260924 16:04:25.390451 4565 leader_election.cc:302] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f86e8e7ee7f348269c2da9a7d18daed5; no voters:
I20260924 16:04:25.391079 4565 leader_election.cc:288] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:25.391203 4570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:25.391894 4570 raft_consensus.cc:686] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [term 1 LEADER]: Becoming Leader. State: Replica: f86e8e7ee7f348269c2da9a7d18daed5, State: Running, Role: LEADER
I20260924 16:04:25.392442 4570 consensus_queue.cc:237] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e8e7ee7f348269c2da9a7d18daed5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41931 } }
I20260924 16:04:25.393177 4565 sys_catalog.cc:564] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:25.395303 4572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f86e8e7ee7f348269c2da9a7d18daed5. Latest consensus state: current_term: 1 leader_uuid: "f86e8e7ee7f348269c2da9a7d18daed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e8e7ee7f348269c2da9a7d18daed5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41931 } } }
I20260924 16:04:25.395792 4572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:25.396539 4571 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f86e8e7ee7f348269c2da9a7d18daed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e8e7ee7f348269c2da9a7d18daed5" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 41931 } } }
I20260924 16:04:25.396865 4571 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:25.398582 4578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:25.403688 4578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:25.413038 4578 catalog_manager.cc:1347] Generated new cluster ID: 48a5117dd3294109827e4ae244b26d68
I20260924 16:04:25.413168 4578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:25.435295 4578 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:25.436604 4578 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:25.445835 4578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5: Generated new TSK 0
I20260924 16:04:25.446744 4578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
Entry for principal kudu/127.16.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab.
I20260924 16:04:25.505936 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41931
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:25.855350 4591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:25.855639 4591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:25.855736 4591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:25.863333 4591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:25.863453 4591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:25.863628 4591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:04:25.873787 4591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.129.keytab
--principal=kudu/127.16.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:25.875249 4591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:25.876598 4591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:25.890427 4596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:25.890471 4599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:25.890426 4597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:25.891273 4591 server_base.cc:1034] running on GCE node
I20260924 16:04:25.891959 4591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:25.892949 4591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:25.894181 4591 hybrid_clock.cc:648] HybridClock initialized: now 1790265865894098 us; error 66 us; skew 500 ppm
I20260924 16:04:25.894574 4591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:25 dist-test-slave-xn4h krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265865, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:25.907687 4591 init.cc:377] Logged in from keytab as kudu/127.16.233.129@KRBTEST.COM (short username kudu)
I20260924 16:04:25.909368 4591 webserver.cc:460] Webserver started at http://127.16.233.129:44173/ using document root <none> and password file <none>
I20260924 16:04:25.910121 4591 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:25.910261 4591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:25.910670 4591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:25.913782 4591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "59c2d0be8be843b18f27d09231cbb901"
format_stamp: "Formatted at 2026-09-24 16:04:25 on dist-test-slave-xn4h"
server_key: "275d915347fa5230dfdc734b3785cc5f"
server_key_iv: "531ed31aa154d1824e07f4961a70abab"
server_key_version: "encryptionkey@0"
I20260924 16:04:25.914644 4591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "59c2d0be8be843b18f27d09231cbb901"
format_stamp: "Formatted at 2026-09-24 16:04:25 on dist-test-slave-xn4h"
server_key: "275d915347fa5230dfdc734b3785cc5f"
server_key_iv: "531ed31aa154d1824e07f4961a70abab"
server_key_version: "encryptionkey@0"
I20260924 16:04:25.919862 4591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:04:25.923274 4606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:25.924578 4591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:04:25.924760 4591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "59c2d0be8be843b18f27d09231cbb901"
format_stamp: "Formatted at 2026-09-24 16:04:25 on dist-test-slave-xn4h"
server_key: "275d915347fa5230dfdc734b3785cc5f"
server_key_iv: "531ed31aa154d1824e07f4961a70abab"
server_key_version: "encryptionkey@0"
I20260924 16:04:25.924973 4591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:25.947788 4591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:25.952951 4591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:25.953258 4591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:25.954248 4591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:25.955639 4591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:25.955742 4591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:25.955891 4591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:25.955971 4591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:25.988828 4591 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:41241
I20260924 16:04:25.988916 4723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:41241 every 8 connection(s)
I20260924 16:04:25.990217 4591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:25.993857 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 4591
I20260924 16:04:25.994071 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:04:25.994710 17318 external_mini_cluster.cc:1428] Setting key 0d77bb796dd0781af5f659611dafe675
Sep 24 16:04:25 dist-test-slave-xn4h krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265865, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.129@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:26.013319 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:25.992817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:47969 (local address 127.16.233.190:41931)
0924 16:04:25.993343 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:25.993347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:25.994591 (+ 1244us) server_negotiation.cc:407] Connection header received
0924 16:04:25.997191 (+ 2600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:25.997203 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:25.997292 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:25.997425 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:25.999196 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.000388 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.001986 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.002808 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.002963 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:26.007135 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:26.007140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:26.007144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:26.007178 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:26.009836 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:26.010881 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:26.010885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:26.010887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:26.011009 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:26.011517 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:26.011520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:26.011521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:26.011780 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:26.011890 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:26.012864 (+ 974us) server_negotiation.cc:299] Negotiation successful
0924 16:04:26.013057 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":292,"threads_started":1}
I20260924 16:04:26.014973 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:25.993358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41931 (local address 127.16.233.129:47969)
0924 16:04:25.994362 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:25.994408 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:25.996899 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:25.997649 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:25.997661 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:25.998227 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:25.999015 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:25.999035 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.000573 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.000576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:26.001835 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:26.001847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.003031 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.003036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:26.003250 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:26.004003 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:26.004028 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:26.006872 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:26.010004 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:26.010016 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:26.010038 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:26.010749 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:26.011125 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:26.011132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:26.011139 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:26.011401 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:26.011907 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:26.011924 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:26.012436 (+ 512us) client_negotiation.cc:769] Sending connection context
0924 16:04:26.012765 (+ 329us) client_negotiation.cc:240] Negotiation successful
0924 16:04:26.013386 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":369,"threads_started":1}
I20260924 16:04:26.016597 4724 heartbeater.cc:347] Connected to a master server at 127.16.233.190:41931
I20260924 16:04:26.016922 4724 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:26.017656 4724 heartbeater.cc:511] Master 127.16.233.190:41931 requested a full tablet report, sending...
I20260924 16:04:26.020195 4530 ts_manager.cc:194] Registered new tserver with Master: 59c2d0be8be843b18f27d09231cbb901 (127.16.233.129:41241)
I20260924 16:04:26.022511 4530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.129@KRBTEST.COM'} at 127.16.233.129:47969
WARNING: no policy specified for kudu/127.16.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
Entry for principal kudu/127.16.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab.
I20260924 16:04:26.054769 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41931
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:26.381762 4730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:26.382055 4730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:26.382223 4730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:26.389360 4730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:26.389496 4730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:26.389675 4730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:04:26.399394 4730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.130.keytab
--principal=kudu/127.16.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:26.400648 4730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:26.401942 4730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:26.414273 4735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:26.414363 4736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:26.414391 4738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:26.415530 4730 server_base.cc:1034] running on GCE node
I20260924 16:04:26.416170 4730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:26.417018 4730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:26.418239 4730 hybrid_clock.cc:648] HybridClock initialized: now 1790265866418198 us; error 52 us; skew 500 ppm
I20260924 16:04:26.418637 4730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:26 dist-test-slave-xn4h krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265866, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:26.421576 4730 init.cc:377] Logged in from keytab as kudu/127.16.233.130@KRBTEST.COM (short username kudu)
I20260924 16:04:26.422979 4730 webserver.cc:460] Webserver started at http://127.16.233.130:45629/ using document root <none> and password file <none>
I20260924 16:04:26.423590 4730 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:26.423684 4730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:26.423930 4730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:26.426687 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "bedb505f7c3c4b20bdbb8289d73901da"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-xn4h"
server_key: "4742417d28ebb6da8d0b972a16264b46"
server_key_iv: "60eec294b8aa16060452d5b2f8375489"
server_key_version: "encryptionkey@0"
I20260924 16:04:26.427377 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "bedb505f7c3c4b20bdbb8289d73901da"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-xn4h"
server_key: "4742417d28ebb6da8d0b972a16264b46"
server_key_iv: "60eec294b8aa16060452d5b2f8375489"
server_key_version: "encryptionkey@0"
I20260924 16:04:26.432214 4730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:04:26.435355 4745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:26.436574 4730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:26.436728 4730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "bedb505f7c3c4b20bdbb8289d73901da"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-xn4h"
server_key: "4742417d28ebb6da8d0b972a16264b46"
server_key_iv: "60eec294b8aa16060452d5b2f8375489"
server_key_version: "encryptionkey@0"
I20260924 16:04:26.436920 4730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:26.454764 4730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:26.458519 4730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:26.458797 4730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:26.459785 4730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:26.461122 4730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:26.461193 4730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:26.461383 4730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:26.461431 4730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:26.494218 4730 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:39609
I20260924 16:04:26.494350 4860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:39609 every 8 connection(s)
I20260924 16:04:26.495618 4730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:26.500452 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 4730
I20260924 16:04:26.500635 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:04:26.501097 17318 external_mini_cluster.cc:1428] Setting key 6d686b5702c19cf0a721bd003c0c616c
Sep 24 16:04:26 dist-test-slave-xn4h krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265866, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.130@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:26.515534 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:26.498140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:57877 (local address 127.16.233.190:41931)
0924 16:04:26.498371 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:26.498378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:26.499623 (+ 1245us) server_negotiation.cc:407] Connection header received
0924 16:04:26.501572 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:26.501577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:26.501666 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:26.501773 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:26.503590 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.504285 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.505866 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.506297 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.506487 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:26.509929 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:26.509933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:26.509937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:26.509975 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:26.512529 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:26.513237 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:26.513243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:26.513246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:26.513377 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:26.513804 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:26.513810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:26.513813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:26.514086 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:26.514254 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:26.514957 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 16:04:26.515192 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:04:26.516512 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:26.498720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41931 (local address 127.16.233.130:57877)
0924 16:04:26.499440 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:26.499511 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:26.501319 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:26.502202 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:26.502210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:26.502874 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:26.503425 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:26.503443 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.504444 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.504449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:26.505638 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:26.505651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.506720 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.506724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:26.506912 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:26.507601 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:26.507627 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:26.509699 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:26.512671 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:26.512680 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:26.512696 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:26.513109 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:26.513497 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:26.513502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:26.513506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:26.513692 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:26.514214 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:26.514223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:26.514608 (+ 385us) client_negotiation.cc:769] Sending connection context
0924 16:04:26.514882 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 16:04:26.515395 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":266,"threads_started":1}
I20260924 16:04:26.517999 4861 heartbeater.cc:347] Connected to a master server at 127.16.233.190:41931
I20260924 16:04:26.518350 4861 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:26.519210 4861 heartbeater.cc:511] Master 127.16.233.190:41931 requested a full tablet report, sending...
I20260924 16:04:26.520860 4530 ts_manager.cc:194] Registered new tserver with Master: bedb505f7c3c4b20bdbb8289d73901da (127.16.233.130:39609)
I20260924 16:04:26.522187 4530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.130@KRBTEST.COM'} at 127.16.233.130:57877
WARNING: no policy specified for kudu/127.16.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
Entry for principal kudu/127.16.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab.
I20260924 16:04:26.557811 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41931
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:26.885919 4866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:26.886260 4866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:26.886485 4866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:26.894273 4866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:26.894472 4866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:26.894681 4866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:04:26.905170 4866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.131.keytab
--principal=kudu/127.16.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:41931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:26.906595 4866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:26.908115 4866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:26.922683 4871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:26.922708 4872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:26.922763 4874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:26.923772 4866 server_base.cc:1034] running on GCE node
I20260924 16:04:26.924429 4866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:26.925420 4866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:26.926646 4866 hybrid_clock.cc:648] HybridClock initialized: now 1790265866926582 us; error 58 us; skew 500 ppm
I20260924 16:04:26.926975 4866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:26 dist-test-slave-xn4h krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265866, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:26.930378 4866 init.cc:377] Logged in from keytab as kudu/127.16.233.131@KRBTEST.COM (short username kudu)
I20260924 16:04:26.931900 4866 webserver.cc:460] Webserver started at http://127.16.233.131:36599/ using document root <none> and password file <none>
I20260924 16:04:26.932618 4866 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:26.932752 4866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:26.933086 4866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:26.936280 4866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "ab4114d4a2be416d87a0781a3f1716dc"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-xn4h"
server_key: "897ef695aa4f8a3d9a6885128003f62b"
server_key_iv: "363d18c435208c03ddebc702608a413d"
server_key_version: "encryptionkey@0"
I20260924 16:04:26.937063 4866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "ab4114d4a2be416d87a0781a3f1716dc"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-xn4h"
server_key: "897ef695aa4f8a3d9a6885128003f62b"
server_key_iv: "363d18c435208c03ddebc702608a413d"
server_key_version: "encryptionkey@0"
I20260924 16:04:26.942616 4866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:04:26.946134 4881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:26.947635 4866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:26.947815 4866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "ab4114d4a2be416d87a0781a3f1716dc"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-xn4h"
server_key: "897ef695aa4f8a3d9a6885128003f62b"
server_key_iv: "363d18c435208c03ddebc702608a413d"
server_key_version: "encryptionkey@0"
I20260924 16:04:26.948033 4866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:26.984237 4866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:26.988368 4866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:26.988686 4866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:26.989715 4866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:26.991132 4866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:26.991240 4866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:26.991394 4866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:26.991465 4866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:27.024359 4866 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:46767
I20260924 16:04:27.024446 4996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:46767 every 8 connection(s)
I20260924 16:04:27.025731 4866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:27.025936 4724 heartbeater.cc:503] Master 127.16.233.190:41931 was elected leader, sending a full tablet report...
I20260924 16:04:27.032856 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 4866
I20260924 16:04:27.033051 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:04:27.033510 17318 external_mini_cluster.cc:1428] Setting key a354dcbf8065a017b042af38aa29dc01
Sep 24 16:04:27 dist-test-slave-xn4h krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265866, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.131@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:27.046837 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.028619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:49239 (local address 127.16.233.190:41931)
0924 16:04:27.029093 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.029098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.030043 (+ 945us) server_negotiation.cc:407] Connection header received
0924 16:04:27.031990 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.031996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.032089 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.032204 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:27.033863 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.034948 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.036621 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.037040 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.037170 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.041109 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:27.041115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:27.041119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:27.041159 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:27.043734 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.044487 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.044494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.044499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.044625 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.045051 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.045057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.045060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.045350 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:27.045503 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.046187 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.046451 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":208,"threads_started":1}
I20260924 16:04:27.047900 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.029198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:41931 (local address 127.16.233.131:49239)
0924 16:04:27.029895 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:27.029935 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:27.031702 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:27.032449 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:27.032457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:27.033034 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:27.033697 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.033716 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.035092 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.035099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.036472 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.036484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.037272 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.037282 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.037643 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.038591 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:27.038626 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:27.040887 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:27.043880 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:27.043889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:27.043906 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:27.044337 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:27.044742 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:27.044747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:27.044751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:27.044939 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:27.045482 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:27.045491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:27.045836 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 16:04:27.046119 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 16:04:27.046691 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":287,"threads_started":1}
I20260924 16:04:27.049381 4997 heartbeater.cc:347] Connected to a master server at 127.16.233.190:41931
I20260924 16:04:27.049698 4997 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:27.050482 4997 heartbeater.cc:511] Master 127.16.233.190:41931 requested a full tablet report, sending...
I20260924 16:04:27.051972 4529 ts_manager.cc:194] Registered new tserver with Master: ab4114d4a2be416d87a0781a3f1716dc (127.16.233.131:46767)
I20260924 16:04:27.052867 4529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.131@KRBTEST.COM'} at 127.16.233.131:49239
I20260924 16:04:27.061163 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:27.077258 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.065178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.16.233.190:41931)
0924 16:04:27.065354 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.065359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.065895 (+ 536us) server_negotiation.cc:407] Connection header received
0924 16:04:27.066047 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.066051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.066163 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.066288 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:27.067738 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.068455 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.069712 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.070154 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.070247 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.072595 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:27.072600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:27.072603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:27.072635 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:27.075085 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.075665 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.075668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.075671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.075780 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.076102 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.076105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.076106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.076407 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:27.076598 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.076854 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.077028 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:04:27.081137 4529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:27.087997 4529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:27.113065 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.100364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39609 (local address 127.0.0.1:36822)
0924 16:04:27.101076 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:27.101126 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:27.101290 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:27.102283 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:27.102297 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:27.102370 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:27.102834 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.102854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.105381 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.105389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.108787 (+ 3398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.108814 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.111929 (+ 3115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.111939 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.112262 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.112334 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 16:04:27.112500 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 16:04:27.112728 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":328,"threads_started":1}
I20260924 16:04:27.114097 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.100540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.16.233.130:39609)
0924 16:04:27.101247 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.101254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.101280 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:04:27.101527 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.101535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.101807 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.102009 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:27.103086 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.105153 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.109117 (+ 3964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.111271 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.111486 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.111743 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.113535 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.113747 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":458,"threads_started":1}
I20260924 16:04:27.118621 4794 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4c0216e0f24dda9e1afd1424a46469 (DEFAULT_TABLE table=test-table [id=969ebebaa500467f9de1b7944991203e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:27.120417 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4c0216e0f24dda9e1afd1424a46469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:27.123502 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.101478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:46767 (local address 127.0.0.1:49514)
0924 16:04:27.102014 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:27.102034 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:27.102140 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:27.113552 (+ 11412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:27.113557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:27.113583 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:27.114173 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.114186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.116147 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.116153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.118404 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.118421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.121150 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.121159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.122537 (+ 1378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.122590 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 16:04:27.122695 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 16:04:27.122898 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":1153,"thread_start_us":261,"threads_started":1}
I20260924 16:04:27.124542 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.101865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.16.233.131:46767)
0924 16:04:27.112585 (+ 10720us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.112595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.112636 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:04:27.112744 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.112752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.113040 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.113248 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:27.114458 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.115960 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.118605 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.120405 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.120711 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.120980 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.123883 (+ 2903us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.124106 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10532,"thread_start_us":10376,"threads_started":1}
I20260924 16:04:27.125049 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.102066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:41241 (local address 127.0.0.1:33320)
0924 16:04:27.110479 (+ 8413us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:27.110506 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:27.110672 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:27.113576 (+ 2904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:27.113580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:27.113611 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:27.114259 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.114273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.119951 (+ 5678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.119957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.122122 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.122138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.124226 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.124233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.124423 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.124458 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:04:27.124540 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 16:04:27.124675 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8182,"mutex_wait_us":95,"thread_start_us":8710,"threads_started":1}
I20260924 16:04:27.127272 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.102586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.16.233.129:41241)
0924 16:04:27.112942 (+ 10356us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.112950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.112976 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:04:27.113068 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.113078 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.113320 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.113577 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:27.115413 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.119755 (+ 4342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.122401 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.123988 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.125608 (+ 1620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.125791 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.125857 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.126027 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10185,"thread_start_us":10046,"threads_started":1}
I20260924 16:04:27.129958 4931 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4c0216e0f24dda9e1afd1424a46469 (DEFAULT_TABLE table=test-table [id=969ebebaa500467f9de1b7944991203e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:27.131649 4931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4c0216e0f24dda9e1afd1424a46469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:27.133656 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4c0216e0f24dda9e1afd1424a46469 (DEFAULT_TABLE table=test-table [id=969ebebaa500467f9de1b7944991203e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:27.135059 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4c0216e0f24dda9e1afd1424a46469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:27.135318 5018 tablet_bootstrap.cc:492] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: Bootstrap starting.
I20260924 16:04:27.138769 5018 tablet_bootstrap.cc:654] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:27.140321 5018 log.cc:824] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:27.144047 5018 tablet_bootstrap.cc:492] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: No bootstrap required, opened a new log
I20260924 16:04:27.144354 5018 ts_tablet_manager.cc:1397] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:04:27.146641 5020 tablet_bootstrap.cc:492] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: Bootstrap starting.
I20260924 16:04:27.150385 5020 tablet_bootstrap.cc:654] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:27.151592 5021 tablet_bootstrap.cc:492] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: Bootstrap starting.
I20260924 16:04:27.152487 5020 log.cc:824] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:27.155320 5021 tablet_bootstrap.cc:654] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:27.155928 5018 raft_consensus.cc:348] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } }
I20260924 16:04:27.156473 5018 raft_consensus.cc:374] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:27.156564 5018 raft_consensus.cc:729] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bedb505f7c3c4b20bdbb8289d73901da, State: Initialized, Role: FOLLOWER
I20260924 16:04:27.156594 5021 log.cc:824] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:27.157310 5018 consensus_queue.cc:260] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } }
I20260924 16:04:27.159102 5018 ts_tablet_manager.cc:1428] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260924 16:04:27.159564 5020 tablet_bootstrap.cc:492] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: No bootstrap required, opened a new log
I20260924 16:04:27.159893 5020 ts_tablet_manager.cc:1397] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260924 16:04:27.162060 5021 tablet_bootstrap.cc:492] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: No bootstrap required, opened a new log
I20260924 16:04:27.162380 5021 ts_tablet_manager.cc:1397] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 16:04:27.162575 4861 heartbeater.cc:503] Master 127.16.233.190:41931 was elected leader, sending a full tablet report...
I20260924 16:04:27.171562 5021 raft_consensus.cc:348] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } }
I20260924 16:04:27.171691 5020 raft_consensus.cc:348] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } }
I20260924 16:04:27.171988 5021 raft_consensus.cc:374] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:27.172075 5020 raft_consensus.cc:374] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:27.172094 5021 raft_consensus.cc:729] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c2d0be8be843b18f27d09231cbb901, State: Initialized, Role: FOLLOWER
I20260924 16:04:27.172183 5020 raft_consensus.cc:729] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4114d4a2be416d87a0781a3f1716dc, State: Initialized, Role: FOLLOWER
I20260924 16:04:27.172767 5021 consensus_queue.cc:260] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } }
I20260924 16:04:27.172765 5020 consensus_queue.cc:260] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } }
I20260924 16:04:27.173867 4997 heartbeater.cc:503] Master 127.16.233.190:41931 was elected leader, sending a full tablet report...
I20260924 16:04:27.174631 5020 ts_tablet_manager.cc:1428] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260924 16:04:27.175817 5021 ts_tablet_manager.cc:1428] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260924 16:04:27.179306 5026 raft_consensus.cc:482] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:27.179581 5026 raft_consensus.cc:504] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } }
I20260924 16:04:27.180989 5026 leader_election.cc:288] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bedb505f7c3c4b20bdbb8289d73901da (127.16.233.130:39609), 59c2d0be8be843b18f27d09231cbb901 (127.16.233.129:41241)
I20260924 16:04:27.187932 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.181383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:51889 (local address 127.16.233.130:39609)
0924 16:04:27.181561 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.181566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.181582 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:04:27.181640 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.181644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.181741 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.181854 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:27.182437 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.183433 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.185884 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.187108 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.187268 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.187344 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.187568 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.187701 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:04:27.188025 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.181196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39609 (local address 127.16.233.131:51889)
0924 16:04:27.181365 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:27.181387 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:27.181492 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:27.182043 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:27.182047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:27.182077 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:27.182285 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.182297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.183568 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.183572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.185720 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.185734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.187258 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.187263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.187403 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.187433 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:04:27.187515 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 16:04:27.187686 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":4}
I20260924 16:04:27.188393 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.181933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:36343 (local address 127.16.233.129:41241)
0924 16:04:27.182088 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.182092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.182173 (+ 81us) server_negotiation.cc:407] Connection header received
0924 16:04:27.182366 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.182373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.182439 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.182531 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:27.183007 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.183867 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.186134 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.187460 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.187622 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.187687 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.188027 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.188177 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 16:04:27.188468 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.181526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:41241 (local address 127.16.233.131:36343)
0924 16:04:27.182084 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:27.182099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:27.182206 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:27.182548 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:27.182551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:27.182570 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:27.182846 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.182854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.184066 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.184072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.185999 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.186009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.187670 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.187676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.187846 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.187886 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:04:27.187980 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:04:27.188115 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":265,"threads_started":1}
I20260924 16:04:27.189306 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4c0216e0f24dda9e1afd1424a46469" candidate_uuid: "ab4114d4a2be416d87a0781a3f1716dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bedb505f7c3c4b20bdbb8289d73901da" is_pre_election: true
I20260924 16:04:27.189652 4677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4c0216e0f24dda9e1afd1424a46469" candidate_uuid: "ab4114d4a2be416d87a0781a3f1716dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c2d0be8be843b18f27d09231cbb901" is_pre_election: true
I20260924 16:04:27.189778 4814 raft_consensus.cc:2454] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab4114d4a2be416d87a0781a3f1716dc in term 0.
I20260924 16:04:27.190150 4677 raft_consensus.cc:2454] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab4114d4a2be416d87a0781a3f1716dc in term 0.
I20260924 16:04:27.190470 4884 leader_election.cc:302] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab4114d4a2be416d87a0781a3f1716dc, bedb505f7c3c4b20bdbb8289d73901da; no voters:
I20260924 16:04:27.190861 5026 raft_consensus.cc:2781] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:27.191016 5026 raft_consensus.cc:482] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:27.191126 5026 raft_consensus.cc:3037] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:27.193337 5026 raft_consensus.cc:504] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } }
I20260924 16:04:27.193988 5026 leader_election.cc:288] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [CANDIDATE]: Term 1 election: Requested vote from peers bedb505f7c3c4b20bdbb8289d73901da (127.16.233.130:39609), 59c2d0be8be843b18f27d09231cbb901 (127.16.233.129:41241)
I20260924 16:04:27.194666 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4c0216e0f24dda9e1afd1424a46469" candidate_uuid: "ab4114d4a2be416d87a0781a3f1716dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bedb505f7c3c4b20bdbb8289d73901da"
I20260924 16:04:27.194670 4677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4c0216e0f24dda9e1afd1424a46469" candidate_uuid: "ab4114d4a2be416d87a0781a3f1716dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c2d0be8be843b18f27d09231cbb901"
I20260924 16:04:27.194886 4814 raft_consensus.cc:3037] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:27.194891 4677 raft_consensus.cc:3037] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:27.198174 4814 raft_consensus.cc:2454] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab4114d4a2be416d87a0781a3f1716dc in term 1.
I20260924 16:04:27.198173 4677 raft_consensus.cc:2454] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab4114d4a2be416d87a0781a3f1716dc in term 1.
I20260924 16:04:27.198616 4884 leader_election.cc:302] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab4114d4a2be416d87a0781a3f1716dc, bedb505f7c3c4b20bdbb8289d73901da; no voters:
I20260924 16:04:27.199048 5026 raft_consensus.cc:2781] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:27.199473 5026 raft_consensus.cc:686] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 1 LEADER]: Becoming Leader. State: Replica: ab4114d4a2be416d87a0781a3f1716dc, State: Running, Role: LEADER
I20260924 16:04:27.200033 5026 consensus_queue.cc:237] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } }
I20260924 16:04:27.204842 4530 catalog_manager.cc:5495] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc reported cstate change: term changed from 0 to 1, leader changed from <none> to ab4114d4a2be416d87a0781a3f1716dc (127.16.233.131). New cstate: current_term: 1 leader_uuid: "ab4114d4a2be416d87a0781a3f1716dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab4114d4a2be416d87a0781a3f1716dc" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 } health_report { overall_health: UNKNOWN } } }
I20260924 16:04:27.226461 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.221101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.16.233.131:46767)
0924 16:04:27.221303 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.221309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.221329 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:04:27.221471 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.221475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.221556 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.221706 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:27.222088 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.222938 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.224314 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.224715 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.224852 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.225350 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:27.225844 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:27.226023 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.226074 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.226159 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:04:27.234872 4677 raft_consensus.cc:1261] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [term 1 FOLLOWER]: Refusing update from remote peer ab4114d4a2be416d87a0781a3f1716dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:27.234872 4814 raft_consensus.cc:1261] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [term 1 FOLLOWER]: Refusing update from remote peer ab4114d4a2be416d87a0781a3f1716dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:27.235826 5028 consensus_queue.cc:1035] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bedb505f7c3c4b20bdbb8289d73901da" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:27.236193 5026 consensus_queue.cc:1035] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "59c2d0be8be843b18f27d09231cbb901" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:04:27.242627 4725 tablet.cc:2369] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:04:27.247588 4862 tablet.cc:2369] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:04:27.247278 5031 mvcc.cc:205] Tried to move back new op lower bound from 7332928992183488512 to 7332928992055394304. Current Snapshot: MvccSnapshot[applied={T|T < 7332928992183488512}]
I20260924 16:04:27.248950 5032 mvcc.cc:205] Tried to move back new op lower bound from 7332928992183488512 to 7332928992055394304. Current Snapshot: MvccSnapshot[applied={T|T < 7332928992183488512}]
I20260924 16:04:27.263360 4529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:27.263696 4529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:27.267418 4529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f86e8e7ee7f348269c2da9a7d18daed5: Sending DeleteTablet for 3 replicas of tablet 0f4c0216e0f24dda9e1afd1424a46469
I20260924 16:04:27.268563 4794 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4c0216e0f24dda9e1afd1424a46469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:27 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:36822
I20260924 16:04:27.268735 4931 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4c0216e0f24dda9e1afd1424a46469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:27 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:49514
I20260924 16:04:27.269042 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4c0216e0f24dda9e1afd1424a46469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:27 UTC) from {username='kudu', principal='kudu/127.16.233.190@KRBTEST.COM'} at 127.0.0.1:33320
I20260924 16:04:27.269666 5040 tablet_replica.cc:330] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: stopping tablet replica
I20260924 16:04:27.269799 5041 tablet_replica.cc:330] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: stopping tablet replica
I20260924 16:04:27.270157 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 4591
I20260924 16:04:27.270222 5041 raft_consensus.cc:2229] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:04:27.270455 5040 raft_consensus.cc:2229] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:27.270889 5041 raft_consensus.cc:2258] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:27.271181 5040 raft_consensus.cc:2258] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:27.273407 5040 ts_tablet_manager.cc:1905] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:27.273547 5041 ts_tablet_manager.cc:1905] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:27.279436 5042 tablet_replica.cc:330] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: stopping tablet replica
I20260924 16:04:27.280007 5042 raft_consensus.cc:2229] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:27.280282 5040 ts_tablet_manager.cc:1918] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:27.280469 5040 log.cc:1196] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/0f4c0216e0f24dda9e1afd1424a46469
I20260924 16:04:27.280602 5042 raft_consensus.cc:2258] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:27.281025 5040 ts_tablet_manager.cc:1939] T 0f4c0216e0f24dda9e1afd1424a46469 P bedb505f7c3c4b20bdbb8289d73901da: Deleting consensus metadata
I20260924 16:04:27.282771 4516 catalog_manager.cc:4841] TS bedb505f7c3c4b20bdbb8289d73901da (127.16.233.130:39609): tablet 0f4c0216e0f24dda9e1afd1424a46469 (table test-table [id=969ebebaa500467f9de1b7944991203e]) successfully deleted
I20260924 16:04:27.283376 5042 ts_tablet_manager.cc:1905] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:27.284389 5041 ts_tablet_manager.cc:1918] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:27.284549 5041 log.cc:1196] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/0f4c0216e0f24dda9e1afd1424a46469
I20260924 16:04:27.285082 5041 ts_tablet_manager.cc:1939] T 0f4c0216e0f24dda9e1afd1424a46469 P ab4114d4a2be416d87a0781a3f1716dc: Deleting consensus metadata
I20260924 16:04:27.286762 4516 catalog_manager.cc:4841] TS ab4114d4a2be416d87a0781a3f1716dc (127.16.233.131:46767): tablet 0f4c0216e0f24dda9e1afd1424a46469 (table test-table [id=969ebebaa500467f9de1b7944991203e]) successfully deleted
I20260924 16:04:27.290560 5042 ts_tablet_manager.cc:1918] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:27.290715 5042 log.cc:1196] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/0f4c0216e0f24dda9e1afd1424a46469
I20260924 16:04:27.291261 5042 ts_tablet_manager.cc:1939] T 0f4c0216e0f24dda9e1afd1424a46469 P 59c2d0be8be843b18f27d09231cbb901: Deleting consensus metadata
I20260924 16:04:27.293154 4516 catalog_manager.cc:4841] TS 59c2d0be8be843b18f27d09231cbb901 (127.16.233.129:41241): tablet 0f4c0216e0f24dda9e1afd1424a46469 (table test-table [id=969ebebaa500467f9de1b7944991203e]) successfully deleted
Sep 24 16:04:27 dist-test-slave-xn4h krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.129@KRBTEST.COM
I20260924 16:04:27.301689 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.271954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.16.233.129:41241)
0924 16:04:27.272127 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.272134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.283716 (+ 11582us) server_negotiation.cc:407] Connection header received
0924 16:04:27.283907 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.283918 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.284005 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.284122 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:27.286033 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.287189 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.289044 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.289742 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.289893 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.295070 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:27.295109 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:27.295127 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:27.295169 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:27.298952 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.299668 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.299679 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.299683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.299856 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.300310 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.300315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.300317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.300673 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:27.300889 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.301227 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.301382 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 16:04:27.302469 4718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:27.423071 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 4591
I20260924 16:04:27.441057 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 4730
Sep 24 16:04:27 dist-test-slave-xn4h krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.130@KRBTEST.COM
I20260924 16:04:27.458132 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.441868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.16.233.130:39609)
0924 16:04:27.442060 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.442066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.442089 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:04:27.442243 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.442252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.442357 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.442461 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:27.443830 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.444983 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.446261 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.446921 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.447055 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.451292 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:27.451311 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:27.451334 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:27.451391 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:27.455272 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.455975 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.455985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.455989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.456142 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.456518 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.456523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.456526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.457007 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:27.457300 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.457531 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.457758 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:04:27.458976 4855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:27.547782 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 4730
I20260924 16:04:27.565747 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 4866
Sep 24 16:04:27 dist-test-slave-xn4h krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.131@KRBTEST.COM
I20260924 16:04:27.582935 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.566700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.16.233.131:46767)
0924 16:04:27.566907 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.566913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.566936 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:04:27.567020 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.567025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.567108 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.567225 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:27.568522 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.569352 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.570798 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.571259 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.571403 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.575764 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:27.575785 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:27.575805 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:27.575879 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:27.579970 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.580644 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.580653 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.580657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.580848 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.581207 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.581213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.581217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.581682 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:27.582012 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.582207 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.582491 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:04:27.583791 4992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:27.707742 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 4866
I20260924 16:04:27.727551 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 4497
I20260924 16:04:27.728644 4560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:27.839556 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 4497
2026-09-24T16:04:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4329 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T16:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:27Z Disabled control of system clock
I20260924 16:04:27.918666 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45545
--webserver_interface=127.16.233.190
--webserver_port=0
--builtin_ntp_servers=127.16.233.148:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:45545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:04:28.255704 5051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:28.256004 5051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:28.256088 5051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:28.263281 5051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:28.263381 5051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:28.263419 5051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:28.263476 5051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:28.263521 5051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:28.272459 5051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:45545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:45545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:28.273713 5051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:28.275420 5051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:28.286893 5059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:28.286859 5057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:28.286911 5056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:28.287950 5051 server_base.cc:1034] running on GCE node
I20260924 16:04:28.288620 5051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:28.289866 5051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:28.291060 5051 hybrid_clock.cc:648] HybridClock initialized: now 1790265868290997 us; error 55 us; skew 500 ppm
I20260924 16:04:28.291394 5051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:28.293452 5051 webserver.cc:460] Webserver started at http://127.16.233.190:43257/ using document root <none> and password file <none>
I20260924 16:04:28.294152 5051 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:28.294235 5051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:28.294574 5051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:28.297282 5051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "f514c781bad64ba686b984999ac32953"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-xn4h"
server_key: "c4bd3a001323bd1cbc89f258e9093c83"
server_key_iv: "99dbadd11248e2b3775d6deec109a67a"
server_key_version: "encryptionkey@0"
I20260924 16:04:28.298007 5051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "f514c781bad64ba686b984999ac32953"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-xn4h"
server_key: "c4bd3a001323bd1cbc89f258e9093c83"
server_key_iv: "99dbadd11248e2b3775d6deec109a67a"
server_key_version: "encryptionkey@0"
I20260924 16:04:28.303125 5051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:04:28.306506 5065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.307809 5051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:04:28.308022 5051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "f514c781bad64ba686b984999ac32953"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-xn4h"
server_key: "c4bd3a001323bd1cbc89f258e9093c83"
server_key_iv: "99dbadd11248e2b3775d6deec109a67a"
server_key_version: "encryptionkey@0"
I20260924 16:04:28.308274 5051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:28.330464 5051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:28.331408 5051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:28.331707 5051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:28.350449 5051 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:45545
I20260924 16:04:28.350466 5118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:45545 every 8 connection(s)
I20260924 16:04:28.351890 5051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:28.355566 5119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:28.361568 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 5051
I20260924 16:04:28.361760 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:28.362181 17318 external_mini_cluster.cc:1428] Setting key ee97102a3909973696a3d872c32316a9
I20260924 16:04:28.366766 5119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953: Bootstrap starting.
I20260924 16:04:28.369748 5119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:28.370792 5119 log.cc:824] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:28.374097 5119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953: No bootstrap required, opened a new log
I20260924 16:04:28.376044 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:28.364146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.16.233.190:45545)
0924 16:04:28.365128 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:28.365139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:28.365178 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:04:28.366904 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:28.366955 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:28.367388 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:28.367753 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:28.368308 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.369545 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.371045 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.371923 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.372156 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:28.373246 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:28.373258 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:28.373277 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:28.373948 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:28.374054 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:28.374089 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:28.374208 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:28.374506 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 16:04:28.374849 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":334,"threads_started":1}
I20260924 16:04:28.382850 5119 raft_consensus.cc:348] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f514c781bad64ba686b984999ac32953" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45545 } }
I20260924 16:04:28.383179 5119 raft_consensus.cc:374] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:28.383252 5119 raft_consensus.cc:729] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f514c781bad64ba686b984999ac32953, State: Initialized, Role: FOLLOWER
I20260924 16:04:28.383828 5119 consensus_queue.cc:260] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f514c781bad64ba686b984999ac32953" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45545 } }
I20260924 16:04:28.384040 5119 raft_consensus.cc:388] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:28.384135 5119 raft_consensus.cc:482] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:28.384264 5119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:28.386418 5119 raft_consensus.cc:504] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f514c781bad64ba686b984999ac32953" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45545 } }
I20260924 16:04:28.386840 5119 leader_election.cc:302] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f514c781bad64ba686b984999ac32953; no voters:
I20260924 16:04:28.387360 5119 leader_election.cc:288] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:28.387550 5124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:28.388280 5124 raft_consensus.cc:686] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [term 1 LEADER]: Becoming Leader. State: Replica: f514c781bad64ba686b984999ac32953, State: Running, Role: LEADER
I20260924 16:04:28.388857 5124 consensus_queue.cc:237] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f514c781bad64ba686b984999ac32953" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45545 } }
I20260924 16:04:28.389562 5119 sys_catalog.cc:564] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:28.391913 5125 sys_catalog.cc:455] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f514c781bad64ba686b984999ac32953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f514c781bad64ba686b984999ac32953" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45545 } } }
I20260924 16:04:28.392441 5125 sys_catalog.cc:458] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:28.392077 5126 sys_catalog.cc:455] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f514c781bad64ba686b984999ac32953. Latest consensus state: current_term: 1 leader_uuid: "f514c781bad64ba686b984999ac32953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f514c781bad64ba686b984999ac32953" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 45545 } } }
I20260924 16:04:28.393106 5126 sys_catalog.cc:458] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:28.393920 5133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:28.399252 5133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:28.407620 5133 catalog_manager.cc:1347] Generated new cluster ID: b8a40c40d6224346994b14c23cef1b4a
I20260924 16:04:28.407732 5133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:28.423900 5133 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:28.425681 5133 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:28.444123 5133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953: Generated new TSK 0
I20260924 16:04:28.445052 5133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:28.459867 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.129:0
--local_ip_for_outbound_sockets=127.16.233.129
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45545
--builtin_ntp_servers=127.16.233.148:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:04:28.782996 5143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:28.783268 5143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:28.783360 5143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:28.790777 5143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:28.790923 5143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:28.791055 5143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.129
I20260924 16:04:28.800240 5143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.129
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.129
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:28.801517 5143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:28.803310 5143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:28.816733 5149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:28.817001 5148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:28.816738 5151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:28.817657 5143 server_base.cc:1034] running on GCE node
I20260924 16:04:28.818387 5143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:28.819366 5143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:28.820636 5143 hybrid_clock.cc:648] HybridClock initialized: now 1790265868820581 us; error 62 us; skew 500 ppm
I20260924 16:04:28.820995 5143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:28.823279 5143 webserver.cc:460] Webserver started at http://127.16.233.129:41357/ using document root <none> and password file <none>
I20260924 16:04:28.824057 5143 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:28.824193 5143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:28.824517 5143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:28.827443 5143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/data/instance:
uuid: "2d339ffc610b44b0934626f7ede2e7cc"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-xn4h"
server_key: "76c54c4c04d49beca08c35023c663b9f"
server_key_iv: "5c6890c5baacfb49c9441059bc5ae73f"
server_key_version: "encryptionkey@0"
I20260924 16:04:28.828253 5143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance:
uuid: "2d339ffc610b44b0934626f7ede2e7cc"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-xn4h"
server_key: "76c54c4c04d49beca08c35023c663b9f"
server_key_iv: "5c6890c5baacfb49c9441059bc5ae73f"
server_key_version: "encryptionkey@0"
I20260924 16:04:28.833384 5143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:04:28.836745 5157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.838023 5143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:04:28.838200 5143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/wal
uuid: "2d339ffc610b44b0934626f7ede2e7cc"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-xn4h"
server_key: "76c54c4c04d49beca08c35023c663b9f"
server_key_iv: "5c6890c5baacfb49c9441059bc5ae73f"
server_key_version: "encryptionkey@0"
I20260924 16:04:28.838490 5143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:28.866191 5143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:28.867115 5143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:28.867429 5143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:28.868466 5143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:28.869956 5143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:28.870078 5143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.870220 5143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:28.870325 5143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.903755 5143 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.129:43303
I20260924 16:04:28.903846 5272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.129:43303 every 8 connection(s)
I20260924 16:04:28.905098 5143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:28.914408 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 5143
I20260924 16:04:28.914580 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/wal/instance
I20260924 16:04:28.915050 17318 external_mini_cluster.cc:1428] Setting key 5cef66662efeb1c68aa61f28164c11b5
I20260924 16:04:28.917436 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:28.907567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:35269 (local address 127.16.233.190:45545)
0924 16:04:28.908028 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:28.908032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:28.909010 (+ 978us) server_negotiation.cc:407] Connection header received
0924 16:04:28.910925 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:28.910931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:28.911032 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:28.911154 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:28.912267 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.913003 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.914665 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.915082 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.915224 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:28.916649 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:28.916654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:28.916657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:28.916674 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:28.916718 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:28.916730 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:28.916828 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:28.917071 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 16:04:28.917177 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":249,"threads_started":1}
I20260924 16:04:28.918172 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.130:0
--local_ip_for_outbound_sockets=127.16.233.130
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45545
--builtin_ntp_servers=127.16.233.148:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:04:28.918857 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:28.907986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45545 (local address 127.16.233.129:35269)
0924 16:04:28.908836 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:28.908876 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:28.910661 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:28.911380 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:28.911387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:28.911470 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:28.912094 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:28.912117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.913178 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.913184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:28.914452 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:28.914464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.915323 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.915331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:28.915563 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:28.916285 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:04:28.916289 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:28.916342 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:04:28.916431 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:28.916833 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:28.916844 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:28.916851 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:04:28.917026 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 16:04:28.917472 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":268,"threads_started":1}
I20260924 16:04:28.920379 5273 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45545
I20260924 16:04:28.920674 5273 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:28.921371 5273 heartbeater.cc:511] Master 127.16.233.190:45545 requested a full tablet report, sending...
I20260924 16:04:28.923607 5083 ts_manager.cc:194] Registered new tserver with Master: 2d339ffc610b44b0934626f7ede2e7cc (127.16.233.129:43303)
I20260924 16:04:28.925773 5083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.129:35269
W20260924 16:04:29.243764 5277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:29.244063 5277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:29.244220 5277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:29.251327 5277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:29.251418 5277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:29.251602 5277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.130
I20260924 16:04:29.260540 5277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.130
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.130
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:29.261794 5277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:29.263621 5277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:29.276297 5285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:29.276264 5283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:29.276610 5282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:29.276961 5277 server_base.cc:1034] running on GCE node
I20260924 16:04:29.277565 5277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:29.278586 5277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:29.279824 5277 hybrid_clock.cc:648] HybridClock initialized: now 1790265869279763 us; error 56 us; skew 500 ppm
I20260924 16:04:29.280146 5277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:29.282280 5277 webserver.cc:460] Webserver started at http://127.16.233.130:42027/ using document root <none> and password file <none>
I20260924 16:04:29.282979 5277 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:29.283120 5277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:29.283422 5277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:29.286191 5277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/data/instance:
uuid: "fd0f613420c142188782d41774cf1c58"
format_stamp: "Formatted at 2026-09-24 16:04:29 on dist-test-slave-xn4h"
server_key: "50d05f88cc5bd37f6e11e60196c77c2e"
server_key_iv: "b7ae3111f9d14c50a4902c107b6bc2b6"
server_key_version: "encryptionkey@0"
I20260924 16:04:29.287042 5277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance:
uuid: "fd0f613420c142188782d41774cf1c58"
format_stamp: "Formatted at 2026-09-24 16:04:29 on dist-test-slave-xn4h"
server_key: "50d05f88cc5bd37f6e11e60196c77c2e"
server_key_iv: "b7ae3111f9d14c50a4902c107b6bc2b6"
server_key_version: "encryptionkey@0"
I20260924 16:04:29.292268 5277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:04:29.295701 5291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.296898 5277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:04:29.297088 5277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/wal
uuid: "fd0f613420c142188782d41774cf1c58"
format_stamp: "Formatted at 2026-09-24 16:04:29 on dist-test-slave-xn4h"
server_key: "50d05f88cc5bd37f6e11e60196c77c2e"
server_key_iv: "b7ae3111f9d14c50a4902c107b6bc2b6"
server_key_version: "encryptionkey@0"
I20260924 16:04:29.297295 5277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:29.320339 5277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:29.321161 5277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:29.321493 5277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:29.322623 5277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:29.324123 5277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:29.324244 5277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.324388 5277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:29.324484 5277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.358085 5277 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.130:39687
I20260924 16:04:29.358145 5405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.130:39687 every 8 connection(s)
I20260924 16:04:29.359385 5277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:29.364050 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 5277
I20260924 16:04:29.364279 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/wal/instance
I20260924 16:04:29.364732 17318 external_mini_cluster.cc:1428] Setting key 7afa75a2e671f955443bcc2bbced5604
I20260924 16:04:29.367780 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.131:0
--local_ip_for_outbound_sockets=127.16.233.131
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45545
--builtin_ntp_servers=127.16.233.148:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:04:29.373167 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.362222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.130:39209 (local address 127.16.233.190:45545)
0924 16:04:29.362443 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.362450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.364023 (+ 1573us) server_negotiation.cc:407] Connection header received
0924 16:04:29.366659 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.366664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.366743 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.366869 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:29.368128 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.368830 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.370426 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.370853 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.370997 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.372369 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:29.372374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:29.372378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:29.372395 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:29.372437 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:29.372449 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:29.372546 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.372764 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.372873 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:04:29.374197 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.362785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45545 (local address 127.16.233.130:39209)
0924 16:04:29.363808 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:29.363865 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:29.366358 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:29.367109 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:29.367118 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:29.367211 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:29.367780 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.367797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.368995 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.369001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.370217 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.370230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.371072 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.371078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.371314 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.372057 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:04:29.372060 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:29.372097 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:04:29.372176 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:29.372545 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:29.372559 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:29.372569 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:04:29.372753 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 16:04:29.373161 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
I20260924 16:04:29.375610 5406 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45545
I20260924 16:04:29.375927 5406 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:29.376638 5406 heartbeater.cc:511] Master 127.16.233.190:45545 requested a full tablet report, sending...
I20260924 16:04:29.378069 5083 ts_manager.cc:194] Registered new tserver with Master: fd0f613420c142188782d41774cf1c58 (127.16.233.130:39687)
I20260924 16:04:29.378901 5083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.130:39209
W20260924 16:04:29.707690 5409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:29.708055 5409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:29.708216 5409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:29.715857 5409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:29.716007 5409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:29.716190 5409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.131
I20260924 16:04:29.728773 5409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.131
--webserver_port=0
--tserver_master_addrs=127.16.233.190:45545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.131
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:29.730350 5409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:29.732152 5409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:29.745388 5415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:29.745466 5417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:29.745466 5414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:29.746622 5409 server_base.cc:1034] running on GCE node
I20260924 16:04:29.747233 5409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:29.748260 5409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:29.749496 5409 hybrid_clock.cc:648] HybridClock initialized: now 1790265869749459 us; error 60 us; skew 500 ppm
I20260924 16:04:29.749825 5409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:04:29.752310 5409 webserver.cc:460] Webserver started at http://127.16.233.131:35385/ using document root <none> and password file <none>
I20260924 16:04:29.753108 5409 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:29.753242 5409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:29.753559 5409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:29.756524 5409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/data/instance:
uuid: "20a572749ab144f59a82ab3f3a730432"
format_stamp: "Formatted at 2026-09-24 16:04:29 on dist-test-slave-xn4h"
server_key: "4671d7bcd65c3b02c677c300c8337e69"
server_key_iv: "45055e6ba41f6be3740809082e1da6ad"
server_key_version: "encryptionkey@0"
I20260924 16:04:29.757350 5409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance:
uuid: "20a572749ab144f59a82ab3f3a730432"
format_stamp: "Formatted at 2026-09-24 16:04:29 on dist-test-slave-xn4h"
server_key: "4671d7bcd65c3b02c677c300c8337e69"
server_key_iv: "45055e6ba41f6be3740809082e1da6ad"
server_key_version: "encryptionkey@0"
I20260924 16:04:29.762610 5409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:04:29.766013 5423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.767374 5409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:29.767555 5409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/wal
uuid: "20a572749ab144f59a82ab3f3a730432"
format_stamp: "Formatted at 2026-09-24 16:04:29 on dist-test-slave-xn4h"
server_key: "4671d7bcd65c3b02c677c300c8337e69"
server_key_iv: "45055e6ba41f6be3740809082e1da6ad"
server_key_version: "encryptionkey@0"
I20260924 16:04:29.767774 5409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:29.785943 5409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:29.786819 5409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:29.787141 5409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:29.788138 5409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:29.789678 5409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:29.789814 5409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.789939 5409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:29.790027 5409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.823978 5409 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.131:39397
I20260924 16:04:29.824071 5539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.131:39397 every 8 connection(s)
I20260924 16:04:29.825389 5409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:29.834554 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 5409
I20260924 16:04:29.834829 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/wal/instance
I20260924 16:04:29.835494 17318 external_mini_cluster.cc:1428] Setting key 6c5bfd96fc761128ec5de92ae2195443
I20260924 16:04:29.837998 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.827973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.131:45187 (local address 127.16.233.190:45545)
0924 16:04:29.828163 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.828168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.829437 (+ 1269us) server_negotiation.cc:407] Connection header received
0924 16:04:29.831255 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.831260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.831333 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.831430 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:29.832434 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.833201 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.834839 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.835459 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.835604 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.837083 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:29.837088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:29.837092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:29.837109 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:29.837142 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:29.837154 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:29.837224 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.837503 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.837601 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:04:29.839155 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.828423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:45545 (local address 127.16.233.131:45187)
0924 16:04:29.829280 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:29.829322 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:29.831006 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:29.831624 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:29.831632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:29.831713 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:29.832284 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.832301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.833356 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.833359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.834652 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.834665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.835690 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.835695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.835886 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.836680 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:04:29.836683 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:29.836735 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:04:29.836850 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:29.837264 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:29.837275 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:29.837283 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:04:29.837483 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 16:04:29.837896 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":264,"threads_started":1}
I20260924 16:04:29.840581 5540 heartbeater.cc:347] Connected to a master server at 127.16.233.190:45545
I20260924 16:04:29.840852 5540 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:29.841572 5540 heartbeater.cc:511] Master 127.16.233.190:45545 requested a full tablet report, sending...
I20260924 16:04:29.843281 5083 ts_manager.cc:194] Registered new tserver with Master: 20a572749ab144f59a82ab3f3a730432 (127.16.233.131:39397)
I20260924 16:04:29.844100 5083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.131:45187
I20260924 16:04:29.851064 17318 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:29.861150 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.855240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.16.233.190:45545)
0924 16:04:29.855411 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.855415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.855964 (+ 549us) server_negotiation.cc:407] Connection header received
0924 16:04:29.856180 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.856186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.856290 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.856416 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:29.857085 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.857849 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.859182 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.859572 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.859662 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.860606 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:29.860610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:29.860613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:29.860629 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:29.860659 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:29.860670 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:29.860737 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.860853 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.860924 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:04:29.865011 5083 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:29.872156 5083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:29.897517 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.884090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:39397 (local address 127.0.0.1:50608)
0924 16:04:29.886157 (+ 2067us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:29.886206 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:29.886479 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:29.888711 (+ 2232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:29.888723 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:29.888767 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:29.889307 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.889325 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.891860 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.891865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.894479 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.894495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.896414 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.896421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.896704 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.896775 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 16:04:29.896930 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 16:04:29.897122 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1700,"mutex_wait_us":49,"thread_start_us":2710,"threads_started":1}
I20260924 16:04:29.897583 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.884444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.16.233.131:39397)
0924 16:04:29.887897 (+ 3453us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.887905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.887940 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:04:29.888048 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.888060 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.888384 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.889611 (+ 1227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:29.889964 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.891711 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.894697 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.896193 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.896391 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.896630 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.896979 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.897176 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3249,"thread_start_us":3086,"threads_started":1}
I20260924 16:04:29.899789 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.887939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39687 (local address 127.0.0.1:47932)
0924 16:04:29.890726 (+ 2787us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:29.890752 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:29.890894 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:29.892152 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:29.892156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:29.892236 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:29.892751 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.892762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.894796 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.894803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.896432 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.896444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.898993 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.898999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.899302 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.899342 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:04:29.899417 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 16:04:29.899508 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2621,"mutex_wait_us":121,"spinlock_wait_cycles":64768,"thread_start_us":2453,"threads_started":1}
I20260924 16:04:29.900101 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.887581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.16.233.130:39687)
0924 16:04:29.891493 (+ 3912us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.891503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.891533 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:04:29.891619 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.891626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.891887 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.892964 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:29.893213 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.894678 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.896663 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.898768 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.899277 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.899481 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.899551 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.899714 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2559,"thread_start_us":2417,"threads_started":1}
I20260924 16:04:29.901723 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.884318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:43303 (local address 127.0.0.1:49072)
0924 16:04:29.887118 (+ 2800us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:29.887140 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:29.887262 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:29.892102 (+ 4840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:29.892106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:29.892142 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:29.892722 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.892740 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.894715 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.894720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.896971 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.896987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.898954 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.898963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.899223 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.901195 (+ 1972us) client_negotiation.cc:769] Sending connection context
0924 16:04:29.901285 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 16:04:29.901413 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2639,"mutex_wait_us":1951,"thread_start_us":1277,"threads_started":1}
I20260924 16:04:29.902567 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.884943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.16.233.129:43303)
0924 16:04:29.891539 (+ 6596us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.891545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.891569 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:04:29.891645 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.891650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.891879 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.892102 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:29.892934 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.894568 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.897196 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.898746 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.898904 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.899121 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.901967 (+ 2846us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.902237 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6435,"thread_start_us":6308,"threads_started":1}
I20260924 16:04:29.902540 5473 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8f068b03884af2b625f3d4db1961d5 (DEFAULT_TABLE table=test-table [id=06eca1efb9744a54b672b371e9536af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:29.904175 5473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8f068b03884af2b625f3d4db1961d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:29.903911 5340 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8f068b03884af2b625f3d4db1961d5 (DEFAULT_TABLE table=test-table [id=06eca1efb9744a54b672b371e9536af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:29.905230 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8f068b03884af2b625f3d4db1961d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:29.906122 5206 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8f068b03884af2b625f3d4db1961d5 (DEFAULT_TABLE table=test-table [id=06eca1efb9744a54b672b371e9536af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:29.907496 5206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8f068b03884af2b625f3d4db1961d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:29.918638 5558 tablet_bootstrap.cc:492] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: Bootstrap starting.
I20260924 16:04:29.921677 5558 tablet_bootstrap.cc:654] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:29.922744 5558 log.cc:824] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:29.925457 5559 tablet_bootstrap.cc:492] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: Bootstrap starting.
I20260924 16:04:29.928974 5561 tablet_bootstrap.cc:492] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: Bootstrap starting.
I20260924 16:04:29.929422 5273 heartbeater.cc:503] Master 127.16.233.190:45545 was elected leader, sending a full tablet report...
I20260924 16:04:29.930606 5559 tablet_bootstrap.cc:654] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:29.931218 5558 tablet_bootstrap.cc:492] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: No bootstrap required, opened a new log
I20260924 16:04:29.931567 5558 ts_tablet_manager.cc:1397] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260924 16:04:29.931912 5559 log.cc:824] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:29.932647 5561 tablet_bootstrap.cc:654] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:29.933847 5561 log.cc:824] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:29.940852 5559 tablet_bootstrap.cc:492] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: No bootstrap required, opened a new log
I20260924 16:04:29.941146 5559 ts_tablet_manager.cc:1397] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260924 16:04:29.945066 5561 tablet_bootstrap.cc:492] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: No bootstrap required, opened a new log
I20260924 16:04:29.945354 5561 ts_tablet_manager.cc:1397] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.014s
I20260924 16:04:29.945514 5558 raft_consensus.cc:348] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } }
I20260924 16:04:29.945986 5558 raft_consensus.cc:374] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:29.946084 5558 raft_consensus.cc:729] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a572749ab144f59a82ab3f3a730432, State: Initialized, Role: FOLLOWER
I20260924 16:04:29.946851 5558 consensus_queue.cc:260] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } }
I20260924 16:04:29.947968 5540 heartbeater.cc:503] Master 127.16.233.190:45545 was elected leader, sending a full tablet report...
I20260924 16:04:29.948617 5558 ts_tablet_manager.cc:1428] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260924 16:04:29.949874 5559 raft_consensus.cc:348] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } }
I20260924 16:04:29.950205 5559 raft_consensus.cc:374] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:29.950336 5559 raft_consensus.cc:729] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0f613420c142188782d41774cf1c58, State: Initialized, Role: FOLLOWER
I20260924 16:04:29.950973 5559 consensus_queue.cc:260] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } }
I20260924 16:04:29.952091 5406 heartbeater.cc:503] Master 127.16.233.190:45545 was elected leader, sending a full tablet report...
I20260924 16:04:29.952809 5559 ts_tablet_manager.cc:1428] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260924 16:04:29.955099 5561 raft_consensus.cc:348] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } }
I20260924 16:04:29.955565 5561 raft_consensus.cc:374] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:29.955681 5561 raft_consensus.cc:729] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d339ffc610b44b0934626f7ede2e7cc, State: Initialized, Role: FOLLOWER
I20260924 16:04:29.956266 5561 consensus_queue.cc:260] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } }
I20260924 16:04:29.957885 5561 ts_tablet_manager.cc:1428] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: Time spent starting tablet: real 0.012s user 0.012s sys 0.002s
I20260924 16:04:30.006718 5566 raft_consensus.cc:482] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:30.007113 5566 raft_consensus.cc:504] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } }
I20260924 16:04:30.009167 5566 leader_election.cc:288] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a572749ab144f59a82ab3f3a730432 (127.16.233.131:39397), fd0f613420c142188782d41774cf1c58 (127.16.233.130:39687)
I20260924 16:04:30.017921 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.009716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:40485 (local address 127.16.233.131:39397)
0924 16:04:30.009905 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.009909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.010503 (+ 594us) server_negotiation.cc:407] Connection header received
0924 16:04:30.010713 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.010718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.010808 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.010926 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:30.011750 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.012809 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.015646 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.016845 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.016985 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.017057 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.017543 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.017670 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:04:30.017922 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.010816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.129:47717 (local address 127.16.233.130:39687)
0924 16:04:30.011030 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.011037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.011248 (+ 211us) server_negotiation.cc:407] Connection header received
0924 16:04:30.011452 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.011456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.011525 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.011596 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:30.012120 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.013094 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.015645 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.016824 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.016991 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.017057 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.017543 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.017667 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:04:30.018268 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.009647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.131:39397 (local address 127.16.233.129:40485)
0924 16:04:30.010391 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.010413 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:30.010561 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:30.011083 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:30.011088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:30.011121 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:30.011565 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.011602 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.012933 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.012938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.015424 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.015439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.017039 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.017049 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.017315 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.017405 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 16:04:30.017501 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 16:04:30.017665 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":201,"spinlock_wait_cycles":22272,"thread_start_us":390,"threads_started":1}
I20260924 16:04:30.018417 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.010520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.130:39687 (local address 127.16.233.129:47717)
0924 16:04:30.011146 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.011166 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:30.011336 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:30.011646 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:30.011652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:30.011682 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:30.011974 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.011981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.013220 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.013225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.015424 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.015439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.017074 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.017081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.017317 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.017408 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 16:04:30.017501 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:04:30.017665 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":123,"thread_start_us":321,"threads_started":1}
I20260924 16:04:30.019667 5493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8f068b03884af2b625f3d4db1961d5" candidate_uuid: "2d339ffc610b44b0934626f7ede2e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a572749ab144f59a82ab3f3a730432" is_pre_election: true
I20260924 16:04:30.019667 5360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8f068b03884af2b625f3d4db1961d5" candidate_uuid: "2d339ffc610b44b0934626f7ede2e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd0f613420c142188782d41774cf1c58" is_pre_election: true
I20260924 16:04:30.020267 5360 raft_consensus.cc:2454] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d339ffc610b44b0934626f7ede2e7cc in term 0.
I20260924 16:04:30.020268 5493 raft_consensus.cc:2454] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d339ffc610b44b0934626f7ede2e7cc in term 0.
I20260924 16:04:30.020960 5161 leader_election.cc:302] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d339ffc610b44b0934626f7ede2e7cc, fd0f613420c142188782d41774cf1c58; no voters:
I20260924 16:04:30.021317 5566 raft_consensus.cc:2781] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:30.021478 5566 raft_consensus.cc:482] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:30.021606 5566 raft_consensus.cc:3037] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:30.023828 5566 raft_consensus.cc:504] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } }
I20260924 16:04:30.024398 5566 leader_election.cc:288] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 20a572749ab144f59a82ab3f3a730432 (127.16.233.131:39397), fd0f613420c142188782d41774cf1c58 (127.16.233.130:39687)
I20260924 16:04:30.024854 5493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8f068b03884af2b625f3d4db1961d5" candidate_uuid: "2d339ffc610b44b0934626f7ede2e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a572749ab144f59a82ab3f3a730432"
I20260924 16:04:30.024955 5360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8f068b03884af2b625f3d4db1961d5" candidate_uuid: "2d339ffc610b44b0934626f7ede2e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd0f613420c142188782d41774cf1c58"
I20260924 16:04:30.025049 5493 raft_consensus.cc:3037] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:30.025200 5360 raft_consensus.cc:3037] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:30.027305 5493 raft_consensus.cc:2454] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d339ffc610b44b0934626f7ede2e7cc in term 1.
I20260924 16:04:30.027565 5360 raft_consensus.cc:2454] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d339ffc610b44b0934626f7ede2e7cc in term 1.
I20260924 16:04:30.027774 5161 leader_election.cc:302] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a572749ab144f59a82ab3f3a730432, 2d339ffc610b44b0934626f7ede2e7cc; no voters:
I20260924 16:04:30.028077 5566 raft_consensus.cc:2781] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:30.028514 5566 raft_consensus.cc:686] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 2d339ffc610b44b0934626f7ede2e7cc, State: Running, Role: LEADER
I20260924 16:04:30.028975 5566 consensus_queue.cc:237] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } }
I20260924 16:04:30.033708 5083 catalog_manager.cc:5495] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d339ffc610b44b0934626f7ede2e7cc (127.16.233.129). New cstate: current_term: 1 leader_uuid: "2d339ffc610b44b0934626f7ede2e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d339ffc610b44b0934626f7ede2e7cc" member_type: VOTER last_known_addr { host: "127.16.233.129" port: 43303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 } health_report { overall_health: UNKNOWN } } }
I20260924 16:04:30.060935 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.055767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.16.233.129:43303)
0924 16:04:30.055945 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.055949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.055964 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:04:30.056053 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.056057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.056143 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.056262 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:30.056584 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.057305 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.058605 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.059000 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.059270 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.059777 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:30.060303 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:30.060423 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.060529 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.060592 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:04:30.069289 5360 raft_consensus.cc:1261] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [term 1 FOLLOWER]: Refusing update from remote peer 2d339ffc610b44b0934626f7ede2e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:30.069331 5493 raft_consensus.cc:1261] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [term 1 FOLLOWER]: Refusing update from remote peer 2d339ffc610b44b0934626f7ede2e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:30.070223 5566 consensus_queue.cc:1035] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a572749ab144f59a82ab3f3a730432" member_type: VOTER last_known_addr { host: "127.16.233.131" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:30.070580 5569 consensus_queue.cc:1035] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd0f613420c142188782d41774cf1c58" member_type: VOTER last_known_addr { host: "127.16.233.130" port: 39687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:04:30.076972 5541 tablet.cc:2369] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:04:30.081614 5572 mvcc.cc:205] Tried to move back new op lower bound from 7332929003792924672 to 7332929003642621952. Current Snapshot: MvccSnapshot[applied={T|T < 7332929003792924672}]
I20260924 16:04:30.083461 5573 mvcc.cc:205] Tried to move back new op lower bound from 7332929003792924672 to 7332929003642621952. Current Snapshot: MvccSnapshot[applied={T|T < 7332929003792924672}]
I20260924 16:04:30.097050 5082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:30.097370 5082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:30.101313 5082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f514c781bad64ba686b984999ac32953: Sending DeleteTablet for 3 replicas of tablet 1e8f068b03884af2b625f3d4db1961d5
I20260924 16:04:30.102433 5473 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8f068b03884af2b625f3d4db1961d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:30 UTC) from {username='slave'} at 127.0.0.1:50608
I20260924 16:04:30.102453 5206 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8f068b03884af2b625f3d4db1961d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:30 UTC) from {username='slave'} at 127.0.0.1:49072
I20260924 16:04:30.102624 5340 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8f068b03884af2b625f3d4db1961d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:30 UTC) from {username='slave'} at 127.0.0.1:47932
I20260924 16:04:30.105436 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5143
I20260924 16:04:30.113734 5581 tablet_replica.cc:330] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: stopping tablet replica
I20260924 16:04:30.114490 5581 raft_consensus.cc:2229] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:30.115248 5581 raft_consensus.cc:2258] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:30.125634 5581 ts_tablet_manager.cc:1905] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:30.135936 5583 tablet_replica.cc:330] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: stopping tablet replica
I20260924 16:04:30.136437 5583 raft_consensus.cc:2229] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:30.136649 5582 tablet_replica.cc:330] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: stopping tablet replica
I20260924 16:04:30.137020 5583 raft_consensus.cc:2258] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:30.137051 5582 raft_consensus.cc:2229] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:04:30.137763 5582 raft_consensus.cc:2258] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:30.140170 5582 ts_tablet_manager.cc:1905] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:30.142575 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.106572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.16.233.129:43303)
0924 16:04:30.106736 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.106742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.130627 (+ 23885us) server_negotiation.cc:407] Connection header received
0924 16:04:30.131187 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.131194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.131383 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.131471 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:30.132191 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.133434 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.137469 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.138219 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.138356 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.140602 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:30.140614 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:30.140631 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:30.141475 (+ 844us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:30.141544 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:30.141574 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:30.141663 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.141990 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.142118 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:04:30.142779 5581 ts_tablet_manager.cc:1918] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:30.142969 5581 log.cc:1196] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-2/wal/wals/1e8f068b03884af2b625f3d4db1961d5
I20260924 16:04:30.143536 5581 ts_tablet_manager.cc:1939] T 1e8f068b03884af2b625f3d4db1961d5 P 20a572749ab144f59a82ab3f3a730432: Deleting consensus metadata
I20260924 16:04:30.143757 5268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:30.144579 5583 ts_tablet_manager.cc:1905] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:30.145958 5069 catalog_manager.cc:4841] TS 20a572749ab144f59a82ab3f3a730432 (127.16.233.131:39397): tablet 1e8f068b03884af2b625f3d4db1961d5 (table test-table [id=06eca1efb9744a54b672b371e9536af6]) successfully deleted
I20260924 16:04:30.151177 5583 ts_tablet_manager.cc:1918] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:30.151364 5583 log.cc:1196] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-1/wal/wals/1e8f068b03884af2b625f3d4db1961d5
I20260924 16:04:30.151901 5583 ts_tablet_manager.cc:1939] T 1e8f068b03884af2b625f3d4db1961d5 P fd0f613420c142188782d41774cf1c58: Deleting consensus metadata
I20260924 16:04:30.153581 5069 catalog_manager.cc:4841] TS fd0f613420c142188782d41774cf1c58 (127.16.233.130:39687): tablet 1e8f068b03884af2b625f3d4db1961d5 (table test-table [id=06eca1efb9744a54b672b371e9536af6]) successfully deleted
I20260924 16:04:30.251716 5582 ts_tablet_manager.cc:1918] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:30.252029 5582 log.cc:1196] T 1e8f068b03884af2b625f3d4db1961d5 P 2d339ffc610b44b0934626f7ede2e7cc: Deleting WAL directory at /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265545293861-17318-0/minicluster-data/ts-0/wal/wals/1e8f068b03884af2b625f3d4db1961d5
I20260924 16:04:30.252308 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 5143
I20260924 16:04:30.271502 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5277
W20260924 16:04:30.271617 5068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.129:43303 (error 108)
I20260924 16:04:30.273936 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.273394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:43303 (local address 127.0.0.1:49090)
0924 16:04:30.273577 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.273704 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 16:04:30.274214 5068 catalog_manager.cc:4584] TS 2d339ffc610b44b0934626f7ede2e7cc (127.16.233.129:43303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e8f068b03884af2b625f3d4db1961d5: Network error: Client connection negotiation failed: client connection to 127.16.233.129:43303: connect: Connection refused (error 111)
I20260924 16:04:30.279176 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.272625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.16.233.130:39687)
0924 16:04:30.272830 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.272834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.273015 (+ 181us) server_negotiation.cc:407] Connection header received
0924 16:04:30.273234 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.273243 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.273353 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.273501 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:30.274087 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.274998 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.276299 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.276693 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.276839 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.277779 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:30.277788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:30.277801 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:30.278485 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:30.278530 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:30.278551 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:30.278641 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.278752 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.278833 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:04:30.279865 5401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:30.328246 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.327722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:43303 (local address 127.0.0.1:49106)
0924 16:04:30.327977 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.328085 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 16:04:30.329281 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.328961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:43303 (local address 127.0.0.1:49118)
0924 16:04:30.329093 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.329158 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 16:04:30.371315 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 5277
I20260924 16:04:30.388607 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5409
I20260924 16:04:30.389807 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.389260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:43303 (local address 127.0.0.1:49120)
0924 16:04:30.389499 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.389604 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 16:04:30.391016 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.390558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:43303 (local address 127.0.0.1:49132)
0924 16:04:30.390738 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.390862 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:04:30.395665 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.389439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.16.233.131:39397)
0924 16:04:30.389674 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.389712 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.389735 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:04:30.389945 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.389950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.390073 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.390185 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:30.390759 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.391640 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.392826 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.393270 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.393363 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.394437 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:30.394444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:30.394458 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:30.395003 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:30.395046 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:30.395065 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:30.395149 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.395262 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.395347 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:04:30.396261 5535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:30.471807 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.471233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:43303 (local address 127.0.0.1:49144)
0924 16:04:30.471512 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.471626 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 16:04:30.472927 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.472504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.129:43303 (local address 127.0.0.1:49158)
0924 16:04:30.472717 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.472799 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.129:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 16:04:30.485302 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 5409
I20260924 16:04:30.502902 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5051
I20260924 16:04:30.504057 5113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:30.618570 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 5051
2026-09-24T16:04:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2780 ms)
[----------] 6 tests from EncryptionPolicyTest (21656 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:30 dist-test-slave-xn4h krb5kdc[5593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:30 dist-test-slave-xn4h krb5kdc[5593](info): set up 2 sockets
Sep 24 16:04:30 dist-test-slave-xn4h krb5kdc[5593](info): commencing operation
krb5kdc: starting...
W20260924 16:04:31.740387 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:31 dist-test-slave-xn4h krb5kdc[5593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:31.998085 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:32.339548 5607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:32.339951 5607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:32.340119 5607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:32.347477 5607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:32.347642 5607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:32.347754 5607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:32.347864 5607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:32.347949 5607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:32.358093 5607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44603
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:32.359519 5607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:32.360857 5607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:32.372766 5612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:32.372735 5615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:32.372705 5613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:32.373698 5607 server_base.cc:1034] running on GCE node
I20260924 16:04:32.374560 5607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:32.376084 5607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:32.377310 5607 hybrid_clock.cc:648] HybridClock initialized: now 1790265872377254 us; error 56 us; skew 500 ppm
I20260924 16:04:32.377622 5607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:32 dist-test-slave-xn4h krb5kdc[5593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:32.390477 5607 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:32.392045 5607 webserver.cc:460] Webserver started at http://127.0.0.1:42605/ using document root <none> and password file <none>
I20260924 16:04:32.392769 5607 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:32.392925 5607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:32.393278 5607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:32.396396 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "7130d2d0926547afbe0a9eae0069a275"
format_stamp: "Formatted at 2026-09-24 16:04:32 on dist-test-slave-xn4h"
server_key: "db08425623bd0de5d367a8b4e7a8d3ec"
server_key_iv: "7b7b7b0636b354ba17c543dd28038ce5"
server_key_version: "encryptionkey@0"
I20260924 16:04:32.397197 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "7130d2d0926547afbe0a9eae0069a275"
format_stamp: "Formatted at 2026-09-24 16:04:32 on dist-test-slave-xn4h"
server_key: "db08425623bd0de5d367a8b4e7a8d3ec"
server_key_iv: "7b7b7b0636b354ba17c543dd28038ce5"
server_key_version: "encryptionkey@0"
I20260924 16:04:32.402439 5607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:04:32.405696 5622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:32.407022 5607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:32.407202 5607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "7130d2d0926547afbe0a9eae0069a275"
format_stamp: "Formatted at 2026-09-24 16:04:32 on dist-test-slave-xn4h"
server_key: "db08425623bd0de5d367a8b4e7a8d3ec"
server_key_iv: "7b7b7b0636b354ba17c543dd28038ce5"
server_key_version: "encryptionkey@0"
I20260924 16:04:32.407389 5607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:32.419435 5607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:32.423382 5607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:32.423683 5607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:32.442355 5607 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44603
I20260924 16:04:32.442441 5673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44603 every 8 connection(s)
I20260924 16:04:32.443785 5607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:32.447445 5674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:32.453094 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 5607
I20260924 16:04:32.453306 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:32.454003 17318 external_mini_cluster.cc:1428] Setting key f122687c099727cff94d829ecd82f9c6
I20260924 16:04:32.457980 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275: Bootstrap starting.
I20260924 16:04:32.461668 5674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:32.462775 5674 log.cc:824] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:32.466082 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275: No bootstrap required, opened a new log
Sep 24 16:04:32 dist-test-slave-xn4h krb5kdc[5593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:04:32.474625 5674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130d2d0926547afbe0a9eae0069a275" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44603 } }
I20260924 16:04:32.475020 5674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:32.475147 5674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7130d2d0926547afbe0a9eae0069a275, State: Initialized, Role: FOLLOWER
I20260924 16:04:32.475780 5674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130d2d0926547afbe0a9eae0069a275" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44603 } }
I20260924 16:04:32.476030 5674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:32.476187 5674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:32.476378 5674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:32.477684 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:32.456171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59697 (local address 127.0.0.1:44603)
0924 16:04:32.456920 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:32.456930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:32.456976 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:04:32.458716 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:32.458738 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:32.459106 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:32.459417 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:32.461274 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:32.462707 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:32.464211 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:32.464838 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:32.464977 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:32.469758 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:32.469767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:32.469780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:32.469821 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:32.473087 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:32.473923 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:32.473930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:32.473937 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:32.474109 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:32.474667 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:32.474670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:32.474672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:32.475334 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:32.475663 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:32.476179 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 16:04:32.476571 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":345,"threads_started":1}
I20260924 16:04:32.478925 5674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130d2d0926547afbe0a9eae0069a275" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44603 } }
I20260924 16:04:32.479419 5674 leader_election.cc:302] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7130d2d0926547afbe0a9eae0069a275; no voters:
I20260924 16:04:32.480121 5674 leader_election.cc:288] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:32.480219 5679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:32.481794 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:32.481828 5679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [term 1 LEADER]: Becoming Leader. State: Replica: 7130d2d0926547afbe0a9eae0069a275, State: Running, Role: LEADER
I20260924 16:04:32.482403 5679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130d2d0926547afbe0a9eae0069a275" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44603 } }
I20260924 16:04:32.482640 5674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:32.485412 5680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7130d2d0926547afbe0a9eae0069a275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130d2d0926547afbe0a9eae0069a275" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44603 } } }
I20260924 16:04:32.485888 5680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:32.490448 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7130d2d0926547afbe0a9eae0069a275. Latest consensus state: current_term: 1 leader_uuid: "7130d2d0926547afbe0a9eae0069a275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130d2d0926547afbe0a9eae0069a275" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44603 } } }
I20260924 16:04:32.491585 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:32.493742 5685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:32.499217 5685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:32.514699 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:32.494855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58031 (local address 127.0.0.1:44603)
0924 16:04:32.495122 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:32.495132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:32.496081 (+ 949us) server_negotiation.cc:407] Connection header received
0924 16:04:32.496360 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:32.496367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:32.496458 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:32.496570 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:32.498491 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:32.499710 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:32.501520 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:32.502208 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:32.502337 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:32.505589 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:32.505595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:32.505599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:32.505636 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:32.510208 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:32.511588 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:32.511596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:32.511602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:32.511766 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:32.512529 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:32.512537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:32.512543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:32.513003 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:32.513291 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:32.514106 (+ 815us) server_negotiation.cc:299] Negotiation successful
0924 16:04:32.514308 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 16:04:32.517555 5685 catalog_manager.cc:1347] Generated new cluster ID: 311daa713985441db2c43068b1814896
I20260924 16:04:32.517663 5685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:32.527261 5685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:32.528697 5685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:32.535116 5685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7130d2d0926547afbe0a9eae0069a275: Generated new TSK 0
I20260924 16:04:32.535879 5685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:32.548265 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5607
I20260924 16:04:32.549450 5669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:32.639876 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 5607
W20260924 16:04:32.657817 5591 connection.cc:535] client connection to 127.0.0.1:44603 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44603): Connection reset by peer (error 104)
2026-09-24T16:04:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:32 dist-test-slave-xn4h krb5kdc[5707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:32 dist-test-slave-xn4h krb5kdc[5707](info): set up 2 sockets
Sep 24 16:04:32 dist-test-slave-xn4h krb5kdc[5707](info): commencing operation
krb5kdc: starting...
W20260924 16:04:33.751852 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:33 dist-test-slave-xn4h krb5kdc[5707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:34.008960 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:34.342631 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:34.342912 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:34.343094 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:34.350368 5720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:34.350514 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:34.350616 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:34.350731 5720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:34.350819 5720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:34.360694 5720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40493
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:34.362085 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:34.363437 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:34.375303 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:34.375356 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:34.375279 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:34.376335 5720 server_base.cc:1034] running on GCE node
I20260924 16:04:34.377146 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:34.378584 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:34.379801 5720 hybrid_clock.cc:648] HybridClock initialized: now 1790265874379746 us; error 52 us; skew 500 ppm
I20260924 16:04:34.380158 5720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:34 dist-test-slave-xn4h krb5kdc[5707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:34.383575 5720 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:34.384984 5720 webserver.cc:460] Webserver started at http://127.0.0.1:38921/ using document root <none> and password file <none>
I20260924 16:04:34.385667 5720 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:34.385813 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:34.386121 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:34.389119 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "4516d73f8d5f4e019be7b501ef4791dd"
format_stamp: "Formatted at 2026-09-24 16:04:34 on dist-test-slave-xn4h"
server_key: "ccad65c96491094d6f95703b10366347"
server_key_iv: "c71cb7e9a8c6839b23bcc974dc19397b"
server_key_version: "encryptionkey@0"
I20260924 16:04:34.389876 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "4516d73f8d5f4e019be7b501ef4791dd"
format_stamp: "Formatted at 2026-09-24 16:04:34 on dist-test-slave-xn4h"
server_key: "ccad65c96491094d6f95703b10366347"
server_key_iv: "c71cb7e9a8c6839b23bcc974dc19397b"
server_key_version: "encryptionkey@0"
I20260924 16:04:34.395183 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:04:34.398636 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:34.399923 5720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:04:34.400108 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "4516d73f8d5f4e019be7b501ef4791dd"
format_stamp: "Formatted at 2026-09-24 16:04:34 on dist-test-slave-xn4h"
server_key: "ccad65c96491094d6f95703b10366347"
server_key_iv: "c71cb7e9a8c6839b23bcc974dc19397b"
server_key_version: "encryptionkey@0"
I20260924 16:04:34.400303 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:34.416796 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:34.420778 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:34.421068 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:34.439436 5720 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40493
I20260924 16:04:34.439478 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40493 every 8 connection(s)
I20260924 16:04:34.440801 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:34.442909 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 5720
I20260924 16:04:34.443097 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:34.443539 17318 external_mini_cluster.cc:1428] Setting key e6874fe34ebb236745bf5a113a1c496d
I20260924 16:04:34.444696 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:34.458282 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:34.445215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54677 (local address 127.0.0.1:40493)
0924 16:04:34.445956 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:34.445972 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:34.446022 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:04:34.447969 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:34.447996 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:34.448531 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:34.448960 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:34.449460 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:34.451592 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:34.453020 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:34.454047 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:34.454233 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:34.455516 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:34.455528 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:34.455550 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:34.456277 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:34.456373 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:34.456404 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:34.456593 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:34.456834 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 16:04:34.457012 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":312,"threads_started":1}
I20260924 16:04:34.459694 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd: Bootstrap starting.
I20260924 16:04:34.462144 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:34.463120 5789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:34.464103 5789 log.cc:824] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:34.467123 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd: No bootstrap required, opened a new log
I20260924 16:04:34.472868 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:34.466633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52113 (local address 127.0.0.1:40493)
0924 16:04:34.466812 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:34.466819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:34.467266 (+ 447us) server_negotiation.cc:407] Connection header received
0924 16:04:34.467480 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:34.467489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:34.467563 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:34.467666 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:34.468174 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:34.468956 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:34.470354 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:34.470798 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:34.470949 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:34.472122 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:34.472126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:34.472130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:34.472150 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:34.472188 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:34.472204 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:34.472331 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:34.472531 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 16:04:34.472588 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:04:34.475246 5789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4516d73f8d5f4e019be7b501ef4791dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } }
I20260924 16:04:34.475664 5789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:34.475771 5789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4516d73f8d5f4e019be7b501ef4791dd, State: Initialized, Role: FOLLOWER
I20260924 16:04:34.476357 5789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4516d73f8d5f4e019be7b501ef4791dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } }
I20260924 16:04:34.476606 5789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:34.476750 5789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:34.476931 5789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:34.479508 5789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4516d73f8d5f4e019be7b501ef4791dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } }
I20260924 16:04:34.480041 5789 leader_election.cc:302] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4516d73f8d5f4e019be7b501ef4791dd; no voters:
I20260924 16:04:34.480531 5789 leader_election.cc:288] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:34.480677 5799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:34.481360 5799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [term 1 LEADER]: Becoming Leader. State: Replica: 4516d73f8d5f4e019be7b501ef4791dd, State: Running, Role: LEADER
I20260924 16:04:34.481910 5799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4516d73f8d5f4e019be7b501ef4791dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } }
I20260924 16:04:34.482689 5789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:34.484894 5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4516d73f8d5f4e019be7b501ef4791dd. Latest consensus state: current_term: 1 leader_uuid: "4516d73f8d5f4e019be7b501ef4791dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4516d73f8d5f4e019be7b501ef4791dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } } }
I20260924 16:04:34.485359 5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:34.486610 5801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4516d73f8d5f4e019be7b501ef4791dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4516d73f8d5f4e019be7b501ef4791dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } } }
I20260924 16:04:34.486968 5801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:34.488727 5809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:34.494598 5809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:34.504091 5809 catalog_manager.cc:1347] Generated new cluster ID: cfa51fc7b15b46af91e49e58c3d6692b
I20260924 16:04:34.504208 5809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:34.543802 5809 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:34.545303 5809 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:34.557672 5809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4516d73f8d5f4e019be7b501ef4791dd: Generated new TSK 0
I20260924 16:04:34.558701 5809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:34.578465 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5720
W20260924 16:04:34.579555 5784 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54677
W20260924 16:04:34.580327 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5720: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:04:34.580516 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 5720
W20260924 16:04:34.597656 5705 connection.cc:535] client connection to 127.0.0.1:40493 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40493): Connection reset by peer (error 104)
2026-09-24T16:04:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:34 dist-test-slave-xn4h krb5kdc[5821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:34 dist-test-slave-xn4h krb5kdc[5821](info): set up 2 sockets
Sep 24 16:04:34 dist-test-slave-xn4h krb5kdc[5821](info): commencing operation
krb5kdc: starting...
W20260924 16:04:35.700013 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:35 dist-test-slave-xn4h krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:35.957021 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:36.285621 5836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:36.285909 5836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:36.286005 5836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:36.293294 5836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:36.293401 5836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:36.293459 5836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:36.293521 5836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:36.293565 5836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:36.303371 5836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:36.304625 5836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:36.305897 5836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:36.318030 5844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:36.318099 5842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:36.318264 5841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:36.319022 5836 server_base.cc:1034] running on GCE node
I20260924 16:04:36.319868 5836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:36.321278 5836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:36.322507 5836 hybrid_clock.cc:648] HybridClock initialized: now 1790265876322476 us; error 56 us; skew 500 ppm
I20260924 16:04:36.322896 5836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:36 dist-test-slave-xn4h krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:36.326542 5836 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:36.328086 5836 webserver.cc:460] Webserver started at http://127.0.0.1:37521/ using document root <none> and password file <none>
I20260924 16:04:36.328727 5836 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:36.328827 5836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:36.329089 5836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:36.332134 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "a07d14f697bf4b69af2d27748d36e42c"
format_stamp: "Formatted at 2026-09-24 16:04:36 on dist-test-slave-xn4h"
server_key: "79019bf46e61239b3e2bd4a3d4b03074"
server_key_iv: "22a7b5a211bcec282d44403d45cf8a26"
server_key_version: "encryptionkey@0"
I20260924 16:04:36.332845 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "a07d14f697bf4b69af2d27748d36e42c"
format_stamp: "Formatted at 2026-09-24 16:04:36 on dist-test-slave-xn4h"
server_key: "79019bf46e61239b3e2bd4a3d4b03074"
server_key_iv: "22a7b5a211bcec282d44403d45cf8a26"
server_key_version: "encryptionkey@0"
I20260924 16:04:36.337862 5836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:04:36.341378 5851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:36.342713 5836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:04:36.342885 5836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "a07d14f697bf4b69af2d27748d36e42c"
format_stamp: "Formatted at 2026-09-24 16:04:36 on dist-test-slave-xn4h"
server_key: "79019bf46e61239b3e2bd4a3d4b03074"
server_key_iv: "22a7b5a211bcec282d44403d45cf8a26"
server_key_version: "encryptionkey@0"
I20260924 16:04:36.343127 5836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:36.348935 5836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:36.353549 5836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:36.353870 5836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:36.372716 5836 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34531
I20260924 16:04:36.372762 5903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34531 every 8 connection(s)
I20260924 16:04:36.374177 5836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:36.377818 5904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:36.380332 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 5836
I20260924 16:04:36.380514 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:36.380954 17318 external_mini_cluster.cc:1428] Setting key 532bb1de444b09b11401fe89fe9a1a5e
I20260924 16:04:36.388792 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:36.382778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48423 (local address 127.0.0.1:34531)
0924 16:04:36.383598 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:36.383609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:36.383649 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:04:36.385283 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:36.385304 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:36.385643 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:36.385964 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:36.387122 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:36.387128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:36.387141 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:36.387156 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:36.387223 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:36.387244 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:36.387431 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:36.387547 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 16:04:36.387870 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":413,"threads_started":1}
I20260924 16:04:36.390084 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c: Bootstrap starting.
I20260924 16:04:36.392643 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:36.393504 5904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:36.394562 5904 log.cc:824] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:36.397480 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c: No bootstrap required, opened a new log
I20260924 16:04:36.399593 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:36.396983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39485 (local address 127.0.0.1:34531)
0924 16:04:36.397158 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:36.397163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:36.397697 (+ 534us) server_negotiation.cc:407] Connection header received
0924 16:04:36.397839 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:36.397843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:36.397918 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:36.398044 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:36.398948 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:36.398957 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:36.398961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:36.398979 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:36.399025 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:36.399046 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:36.399169 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:36.399228 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 16:04:36.399290 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:04:36.405965 5904 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d14f697bf4b69af2d27748d36e42c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260924 16:04:36.406450 5904 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:36.406564 5904 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07d14f697bf4b69af2d27748d36e42c, State: Initialized, Role: FOLLOWER
I20260924 16:04:36.407050 5904 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d14f697bf4b69af2d27748d36e42c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260924 16:04:36.407269 5904 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:36.407366 5904 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:36.407549 5904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:36.410023 5904 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d14f697bf4b69af2d27748d36e42c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260924 16:04:36.410547 5904 leader_election.cc:302] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07d14f697bf4b69af2d27748d36e42c; no voters:
I20260924 16:04:36.411051 5904 leader_election.cc:288] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:36.411196 5915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:36.411850 5915 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [term 1 LEADER]: Becoming Leader. State: Replica: a07d14f697bf4b69af2d27748d36e42c, State: Running, Role: LEADER
I20260924 16:04:36.412336 5915 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d14f697bf4b69af2d27748d36e42c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260924 16:04:36.413054 5904 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:36.415380 5916 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07d14f697bf4b69af2d27748d36e42c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d14f697bf4b69af2d27748d36e42c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } } }
I20260924 16:04:36.415335 5917 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07d14f697bf4b69af2d27748d36e42c. Latest consensus state: current_term: 1 leader_uuid: "a07d14f697bf4b69af2d27748d36e42c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d14f697bf4b69af2d27748d36e42c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } } }
I20260924 16:04:36.415779 5916 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:36.415849 5917 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:36.419312 5923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:36.424875 5923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:36.434172 5923 catalog_manager.cc:1347] Generated new cluster ID: acf859c6e1c942d7b6ef6d0ec35721fb
I20260924 16:04:36.434329 5923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:36.471187 5923 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:36.471666 5923 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:36.483785 5923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07d14f697bf4b69af2d27748d36e42c: Generated new TSK 0
I20260924 16:04:36.484617 5923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:36.494831 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5836
W20260924 16:04:36.495796 5899 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48423
W20260924 16:04:36.496482 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5836: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:04:36.496615 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 5836
2026-09-24T16:04:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:36 dist-test-slave-xn4h krb5kdc[5936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:36 dist-test-slave-xn4h krb5kdc[5936](info): set up 2 sockets
Sep 24 16:04:36 dist-test-slave-xn4h krb5kdc[5936](info): commencing operation
krb5kdc: starting...
W20260924 16:04:37.604019 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:37 dist-test-slave-xn4h krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:37.851616 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:38.183996 5949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:38.184319 5949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:38.184494 5949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:38.191581 5949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:38.191689 5949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:38.191792 5949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:38.191915 5949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:38.191975 5949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:38.201712 5949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:38.203040 5949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:38.204358 5949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:38.216162 5955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:38.216238 5956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:38.216274 5958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:38.217092 5949 server_base.cc:1034] running on GCE node
I20260924 16:04:38.217885 5949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:38.219316 5949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:38.220551 5949 hybrid_clock.cc:648] HybridClock initialized: now 1790265878220461 us; error 78 us; skew 500 ppm
I20260924 16:04:38.220925 5949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:38 dist-test-slave-xn4h krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:38.234593 5949 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:38.236287 5949 webserver.cc:460] Webserver started at http://127.0.0.1:36199/ using document root <none> and password file <none>
I20260924 16:04:38.237022 5949 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:38.237180 5949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:38.237517 5949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:38.240680 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "be100a1b0c94443fba3cac0c1ae5336f"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-xn4h"
server_key: "1a3c2377cb8f2870b216ddad56b9b2a2"
server_key_iv: "dd48b57e799c9b82ce4c4b9372b926ec"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.241490 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "be100a1b0c94443fba3cac0c1ae5336f"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-xn4h"
server_key: "1a3c2377cb8f2870b216ddad56b9b2a2"
server_key_iv: "dd48b57e799c9b82ce4c4b9372b926ec"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.246846 5949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:04:38.250213 5966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.251514 5949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 16:04:38.251698 5949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "be100a1b0c94443fba3cac0c1ae5336f"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-xn4h"
server_key: "1a3c2377cb8f2870b216ddad56b9b2a2"
server_key_iv: "dd48b57e799c9b82ce4c4b9372b926ec"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.251894 5949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:38.267318 5949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:38.271620 5949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:38.271920 5949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:38.290159 5949 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44777
I20260924 16:04:38.290244 6017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44777 every 8 connection(s)
I20260924 16:04:38.291587 5949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:38.295368 6019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:38.296284 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 5949
I20260924 16:04:38.296475 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:38.296900 17318 external_mini_cluster.cc:1428] Setting key 3016095de1a5025a983cf7877c939888
I20260924 16:04:38.309806 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f: Bootstrap starting.
Sep 24 16:04:38 dist-test-slave-xn4h krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:04:38.313063 6019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:38.314495 6019 log.cc:824] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:38.317888 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f: No bootstrap required, opened a new log
I20260924 16:04:38.321431 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.298740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55271 (local address 127.0.0.1:44777)
0924 16:04:38.299555 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.299570 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.299620 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:04:38.302111 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.302140 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.302688 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.303146 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:38.304459 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.306521 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.307777 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.308816 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.309008 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.313222 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:38.313235 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:38.313258 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:38.313313 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:38.317292 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.318070 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.318080 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.318092 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.318250 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.318688 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.318696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.318703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.319218 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:38.319545 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.320045 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.320448 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":323,"threads_started":1}
I20260924 16:04:38.325255 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:38.326774 6019 raft_consensus.cc:348] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be100a1b0c94443fba3cac0c1ae5336f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44777 } }
I20260924 16:04:38.327145 6019 raft_consensus.cc:374] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:38.327252 6019 raft_consensus.cc:729] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be100a1b0c94443fba3cac0c1ae5336f, State: Initialized, Role: FOLLOWER
I20260924 16:04:38.327874 6019 consensus_queue.cc:260] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be100a1b0c94443fba3cac0c1ae5336f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44777 } }
I20260924 16:04:38.328150 6019 raft_consensus.cc:388] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:38.328300 6019 raft_consensus.cc:482] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:38.328498 6019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:38.331429 6019 raft_consensus.cc:504] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be100a1b0c94443fba3cac0c1ae5336f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44777 } }
I20260924 16:04:38.331933 6019 leader_election.cc:302] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be100a1b0c94443fba3cac0c1ae5336f; no voters:
I20260924 16:04:38.332553 6019 leader_election.cc:288] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:38.332691 6030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:38.335198 6019 sys_catalog.cc:564] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:38.339376 6030 raft_consensus.cc:686] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [term 1 LEADER]: Becoming Leader. State: Replica: be100a1b0c94443fba3cac0c1ae5336f, State: Running, Role: LEADER
I20260924 16:04:38.340161 6030 consensus_queue.cc:237] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be100a1b0c94443fba3cac0c1ae5336f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44777 } }
I20260924 16:04:38.343350 6031 sys_catalog.cc:455] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be100a1b0c94443fba3cac0c1ae5336f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be100a1b0c94443fba3cac0c1ae5336f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44777 } } }
I20260924 16:04:38.343746 6031 sys_catalog.cc:458] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:38.346015 6041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:38.346051 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.329911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50671 (local address 127.0.0.1:44777)
0924 16:04:38.330147 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.330153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.330517 (+ 364us) server_negotiation.cc:407] Connection header received
0924 16:04:38.330695 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.330698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.330778 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.330870 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:38.332513 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.333666 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.334950 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.335500 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.335630 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.338214 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:38.338224 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:38.338227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:38.338257 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:38.341545 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.342453 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.342462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.342469 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.342661 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.343204 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.343211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.343216 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.343710 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:38.343982 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.345399 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.345661 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:04:38.348451 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [sys.catalog]: SysCatalogTable state changed. Reason: New leader be100a1b0c94443fba3cac0c1ae5336f. Latest consensus state: current_term: 1 leader_uuid: "be100a1b0c94443fba3cac0c1ae5336f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be100a1b0c94443fba3cac0c1ae5336f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44777 } } }
I20260924 16:04:38.348997 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:38.354705 6041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:38.366283 6041 catalog_manager.cc:1347] Generated new cluster ID: b4bdb5313f1c4b2b86cfc4c54b4c858a
I20260924 16:04:38.366438 6041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:38.396847 6041 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:38.398823 6041 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:38.407982 6041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be100a1b0c94443fba3cac0c1ae5336f: Generated new TSK 0
I20260924 16:04:38.409018 6041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:38.419934 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 5949
I20260924 16:04:38.421828 6013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:38.515260 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 5949
2026-09-24T16:04:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:38 dist-test-slave-xn4h krb5kdc[6052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:38 dist-test-slave-xn4h krb5kdc[6052](info): set up 2 sockets
Sep 24 16:04:38 dist-test-slave-xn4h krb5kdc[6052](info): commencing operation
krb5kdc: starting...
W20260924 16:04:39.636384 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:39 dist-test-slave-xn4h krb5kdc[6052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:39.899670 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:40.244191 6065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:40.244498 6065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:40.244596 6065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:40.251893 6065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:40.252028 6065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:40.252077 6065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:40.252134 6065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:40.252178 6065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:40.262293 6065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:40.263666 6065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:40.265019 6065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:40.276670 6071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:40.276598 6074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:40.276643 6072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:40.277873 6065 server_base.cc:1034] running on GCE node
I20260924 16:04:40.278666 6065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:40.280078 6065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:40.281286 6065 hybrid_clock.cc:648] HybridClock initialized: now 1790265880281247 us; error 26 us; skew 500 ppm
I20260924 16:04:40.281646 6065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:40 dist-test-slave-xn4h krb5kdc[6052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:40.284823 6065 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:40.286474 6065 webserver.cc:460] Webserver started at http://127.0.0.1:46225/ using document root <none> and password file <none>
I20260924 16:04:40.287174 6065 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:40.287323 6065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:40.287634 6065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:40.290763 6065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "5d50db01109e47b6abbeea6cda426956"
format_stamp: "Formatted at 2026-09-24 16:04:40 on dist-test-slave-xn4h"
server_key: "e69e102516379615c4bcce59848eb334"
server_key_iv: "47e8efdf639578ebac1de9164cc6e18d"
server_key_version: "encryptionkey@0"
I20260924 16:04:40.291571 6065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "5d50db01109e47b6abbeea6cda426956"
format_stamp: "Formatted at 2026-09-24 16:04:40 on dist-test-slave-xn4h"
server_key: "e69e102516379615c4bcce59848eb334"
server_key_iv: "47e8efdf639578ebac1de9164cc6e18d"
server_key_version: "encryptionkey@0"
I20260924 16:04:40.297029 6065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 16:04:40.300704 6081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:40.302184 6065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:04:40.302441 6065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "5d50db01109e47b6abbeea6cda426956"
format_stamp: "Formatted at 2026-09-24 16:04:40 on dist-test-slave-xn4h"
server_key: "e69e102516379615c4bcce59848eb334"
server_key_iv: "47e8efdf639578ebac1de9164cc6e18d"
server_key_version: "encryptionkey@0"
I20260924 16:04:40.302667 6065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:40.315377 6065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:40.319420 6065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:40.319718 6065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:40.338984 6065 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41871
I20260924 16:04:40.338996 6133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41871 every 8 connection(s)
I20260924 16:04:40.340415 6065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:40.344168 6134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:40.345693 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 6065
I20260924 16:04:40.345898 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:40.346405 17318 external_mini_cluster.cc:1428] Setting key ccb43a0f3c1dbc3fee96e473aea4991e
I20260924 16:04:40.358296 6134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956: Bootstrap starting.
I20260924 16:04:40.360388 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.348357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47355 (local address 127.0.0.1:41871)
0924 16:04:40.349213 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:40.349225 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:40.349266 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:04:40.350954 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:40.350983 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:40.351404 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:40.351717 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:40.352363 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:40.353651 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:40.355679 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:40.356421 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:40.356557 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:40.357888 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:40.357897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:40.357918 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:40.358628 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:40.358709 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:40.358736 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:40.358832 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:40.359060 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 16:04:40.359355 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":415,"threads_started":1}
I20260924 16:04:40.361932 6134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:40.363271 6134 log.cc:824] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:40.364568 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:40.366722 6134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956: No bootstrap required, opened a new log
I20260924 16:04:40.376744 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.369930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38303 (local address 127.0.0.1:41871)
0924 16:04:40.370196 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:40.370206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:40.370863 (+ 657us) server_negotiation.cc:407] Connection header received
0924 16:04:40.371024 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:40.371031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:40.371121 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:40.371241 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:40.371722 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:40.372864 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:40.374101 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:40.374757 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:40.374857 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:40.376027 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:40.376032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:40.376036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:40.376053 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:40.376092 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:40.376107 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:40.376171 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:40.376334 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 16:04:40.376406 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:04:40.376865 6134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d50db01109e47b6abbeea6cda426956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41871 } }
I20260924 16:04:40.377338 6134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:40.377468 6134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d50db01109e47b6abbeea6cda426956, State: Initialized, Role: FOLLOWER
I20260924 16:04:40.378070 6134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d50db01109e47b6abbeea6cda426956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41871 } }
I20260924 16:04:40.378337 6134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:40.378500 6134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:40.378686 6134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:40.380877 6134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d50db01109e47b6abbeea6cda426956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41871 } }
I20260924 16:04:40.381389 6134 leader_election.cc:302] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d50db01109e47b6abbeea6cda426956; no voters:
I20260924 16:04:40.382031 6134 leader_election.cc:288] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:40.382170 6144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:40.383265 6144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [term 1 LEADER]: Becoming Leader. State: Replica: 5d50db01109e47b6abbeea6cda426956, State: Running, Role: LEADER
I20260924 16:04:40.383759 6144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d50db01109e47b6abbeea6cda426956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41871 } }
I20260924 16:04:40.384212 6134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:40.386495 6148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d50db01109e47b6abbeea6cda426956. Latest consensus state: current_term: 1 leader_uuid: "5d50db01109e47b6abbeea6cda426956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d50db01109e47b6abbeea6cda426956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41871 } } }
I20260924 16:04:40.387310 6147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d50db01109e47b6abbeea6cda426956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d50db01109e47b6abbeea6cda426956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41871 } } }
I20260924 16:04:40.387773 6148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:40.387905 6147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:40.391304 6154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:40.396638 6154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:40.407276 6154 catalog_manager.cc:1347] Generated new cluster ID: f6d7974b49374f4299aba6072cfcaa5b
I20260924 16:04:40.407411 6154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:40.436672 6154 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:40.438210 6154 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:40.447077 6154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d50db01109e47b6abbeea6cda426956: Generated new TSK 0
I20260924 16:04:40.448117 6154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:40.455932 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6065
W20260924 16:04:40.457024 6129 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47355
W20260924 16:04:40.457711 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6065: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:04:40.457850 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 6065
2026-09-24T16:04:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:40 dist-test-slave-xn4h krb5kdc[6167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:40 dist-test-slave-xn4h krb5kdc[6167](info): set up 2 sockets
Sep 24 16:04:40 dist-test-slave-xn4h krb5kdc[6167](info): commencing operation
krb5kdc: starting...
W20260924 16:04:41.572170 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:41 dist-test-slave-xn4h krb5kdc[6167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:41.825111 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:42.159446 6180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:42.159722 6180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:42.159907 6180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:42.167126 6180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:42.167263 6180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:42.167372 6180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:42.167479 6180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:42.167567 6180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:42.177413 6180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:42.178834 6180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:42.180248 6180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:42.192245 6186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:42.192322 6188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:42.192282 6185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:42.193459 6180 server_base.cc:1034] running on GCE node
I20260924 16:04:42.194217 6180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:42.195699 6180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:42.196941 6180 hybrid_clock.cc:648] HybridClock initialized: now 1790265882196897 us; error 56 us; skew 500 ppm
I20260924 16:04:42.197368 6180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:42 dist-test-slave-xn4h krb5kdc[6167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:42.200508 6180 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:42.202021 6180 webserver.cc:460] Webserver started at http://127.0.0.1:40229/ using document root <none> and password file <none>
I20260924 16:04:42.202781 6180 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:42.202929 6180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:42.203246 6180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:42.206293 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "52a422d9b53341c996803db879e4d25f"
format_stamp: "Formatted at 2026-09-24 16:04:42 on dist-test-slave-xn4h"
server_key: "ec6841d027396263cc57580518a97e20"
server_key_iv: "828171cadd2fa372d1fc8f2218a48c11"
server_key_version: "encryptionkey@0"
I20260924 16:04:42.207158 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "52a422d9b53341c996803db879e4d25f"
format_stamp: "Formatted at 2026-09-24 16:04:42 on dist-test-slave-xn4h"
server_key: "ec6841d027396263cc57580518a97e20"
server_key_iv: "828171cadd2fa372d1fc8f2218a48c11"
server_key_version: "encryptionkey@0"
I20260924 16:04:42.212276 6180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 16:04:42.215874 6195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:42.217139 6180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:04:42.217327 6180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "52a422d9b53341c996803db879e4d25f"
format_stamp: "Formatted at 2026-09-24 16:04:42 on dist-test-slave-xn4h"
server_key: "ec6841d027396263cc57580518a97e20"
server_key_iv: "828171cadd2fa372d1fc8f2218a48c11"
server_key_version: "encryptionkey@0"
I20260924 16:04:42.217541 6180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:42.223464 6180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:42.227442 6180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:42.227774 6180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:42.246567 6180 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38295
I20260924 16:04:42.246589 6248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38295 every 8 connection(s)
I20260924 16:04:42.248000 6180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:42.250506 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 6180
I20260924 16:04:42.250676 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:42.251248 17318 external_mini_cluster.cc:1428] Setting key c6426bfa0d134849e67d722f3283540a
I20260924 16:04:42.252324 6249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:42.259002 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:42.253045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37331 (local address 127.0.0.1:38295)
0924 16:04:42.253858 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:42.253871 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:42.253913 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:04:42.255548 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:42.255568 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:42.255914 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:42.256256 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:42.257339 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:42.257344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:42.257365 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:42.257379 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:42.257461 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:42.257481 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:42.257647 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:42.257747 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 16:04:42.257962 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":380,"threads_started":1}
I20260924 16:04:42.262897 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:42.265725 6249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f: Bootstrap starting.
I20260924 16:04:42.269434 6249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:42.270008 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:42.267345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56883 (local address 127.0.0.1:38295)
0924 16:04:42.267558 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:42.267565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:42.267921 (+ 356us) server_negotiation.cc:407] Connection header received
0924 16:04:42.268111 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:42.268116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:42.268170 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:42.268268 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:42.269309 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:42.269315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:42.269323 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:42.269342 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:42.269393 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:42.269413 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:42.269534 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:42.269661 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 16:04:42.269724 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:04:42.270788 6249 log.cc:824] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:42.273478 6249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f: No bootstrap required, opened a new log
I20260924 16:04:42.280639 6249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a422d9b53341c996803db879e4d25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260924 16:04:42.281046 6249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:42.281149 6249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a422d9b53341c996803db879e4d25f, State: Initialized, Role: FOLLOWER
I20260924 16:04:42.281757 6249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a422d9b53341c996803db879e4d25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260924 16:04:42.282019 6249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:42.282161 6249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:42.282349 6249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:42.284487 6249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a422d9b53341c996803db879e4d25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260924 16:04:42.284993 6249 leader_election.cc:302] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52a422d9b53341c996803db879e4d25f; no voters:
I20260924 16:04:42.285717 6249 leader_election.cc:288] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:42.285847 6259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:42.286546 6259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [term 1 LEADER]: Becoming Leader. State: Replica: 52a422d9b53341c996803db879e4d25f, State: Running, Role: LEADER
I20260924 16:04:42.287195 6259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a422d9b53341c996803db879e4d25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260924 16:04:42.288148 6249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:42.290122 6260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52a422d9b53341c996803db879e4d25f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a422d9b53341c996803db879e4d25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260924 16:04:42.290233 6261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52a422d9b53341c996803db879e4d25f. Latest consensus state: current_term: 1 leader_uuid: "52a422d9b53341c996803db879e4d25f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a422d9b53341c996803db879e4d25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260924 16:04:42.290706 6261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:42.290578 6260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:42.296078 6265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:42.301560 6265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:42.311424 6265 catalog_manager.cc:1347] Generated new cluster ID: 29aa9f0a73c94caf9372e208da91e9dc
I20260924 16:04:42.311568 6265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:42.322649 6265 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:42.323107 6265 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:42.331892 6265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52a422d9b53341c996803db879e4d25f: Generated new TSK 0
I20260924 16:04:42.332703 6265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:42.348845 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6180
W20260924 16:04:42.350255 6244 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37331
W20260924 16:04:42.351037 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6180: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:04:42.351181 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 6180
2026-09-24T16:04:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1892 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:42 dist-test-slave-xn4h krb5kdc[6280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:42 dist-test-slave-xn4h krb5kdc[6280](info): set up 2 sockets
Sep 24 16:04:42 dist-test-slave-xn4h krb5kdc[6280](info): commencing operation
krb5kdc: starting...
W20260924 16:04:43.452494 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:43 dist-test-slave-xn4h krb5kdc[6280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:43.723676 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:44.063687 6294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:44.064002 6294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:44.064095 6294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:44.071655 6294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:44.071789 6294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:44.071842 6294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:44.071906 6294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:44.071949 6294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:44.082131 6294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:44.083582 6294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:44.084879 6294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:44.096812 6299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:44.096851 6300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:44.096879 6302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:44.098035 6294 server_base.cc:1034] running on GCE node
I20260924 16:04:44.098956 6294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:44.100497 6294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:44.101737 6294 hybrid_clock.cc:648] HybridClock initialized: now 1790265884101682 us; error 50 us; skew 500 ppm
I20260924 16:04:44.102146 6294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:44 dist-test-slave-xn4h krb5kdc[6280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:44.112004 6294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:44.113855 6294 webserver.cc:460] Webserver started at http://127.0.0.1:34287/ using document root <none> and password file <none>
I20260924 16:04:44.114687 6294 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:44.114846 6294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:44.115206 6294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:44.119422 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "414d66e80a1c4dfebecafc2af41c1d37"
format_stamp: "Formatted at 2026-09-24 16:04:44 on dist-test-slave-xn4h"
server_key: "ecd29dd080438920fed7a588c575ae21"
server_key_iv: "a636b2b154f86639f838d54266253712"
server_key_version: "encryptionkey@0"
I20260924 16:04:44.120594 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "414d66e80a1c4dfebecafc2af41c1d37"
format_stamp: "Formatted at 2026-09-24 16:04:44 on dist-test-slave-xn4h"
server_key: "ecd29dd080438920fed7a588c575ae21"
server_key_iv: "a636b2b154f86639f838d54266253712"
server_key_version: "encryptionkey@0"
I20260924 16:04:44.127255 6294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260924 16:04:44.130873 6309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:44.132565 6294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 16:04:44.132795 6294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "414d66e80a1c4dfebecafc2af41c1d37"
format_stamp: "Formatted at 2026-09-24 16:04:44 on dist-test-slave-xn4h"
server_key: "ecd29dd080438920fed7a588c575ae21"
server_key_iv: "a636b2b154f86639f838d54266253712"
server_key_version: "encryptionkey@0"
I20260924 16:04:44.133044 6294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:44.148640 6294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:44.152936 6294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:44.153246 6294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:44.172254 6294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37207
I20260924 16:04:44.172283 6360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37207 every 8 connection(s)
I20260924 16:04:44.173724 6294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:44.177593 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:44.180197 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 6294
I20260924 16:04:44.180397 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:44.180924 17318 external_mini_cluster.cc:1428] Setting key c6f8b7faaa69a30ad4fd8fa2ef5f840b
I20260924 16:04:44.190001 6361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37: Bootstrap starting.
I20260924 16:04:44.193360 6361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:44.194502 6361 log.cc:824] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:04:44 dist-test-slave-xn4h krb5kdc[6280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:04:44.197721 6361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37: No bootstrap required, opened a new log
I20260924 16:04:44.203920 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:44.183035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:35027 (local address 127.0.0.1:37207)
0924 16:04:44.183845 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:44.183860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:44.183913 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:04:44.185658 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:44.185681 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:44.186098 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:44.186518 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:44.188100 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:44.189386 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:44.190789 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:44.191486 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:44.191725 (+ 239us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:44.196267 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:44.196276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:44.196291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:44.196329 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:44.199418 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:44.200449 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:44.200459 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:44.200465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:44.200627 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:44.201142 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:44.201146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:44.201149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:44.201690 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:44.201956 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:44.202476 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 16:04:44.202842 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":345,"threads_started":1}
I20260924 16:04:44.206797 6361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414d66e80a1c4dfebecafc2af41c1d37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37207 } }
I20260924 16:04:44.207226 6361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:44.207361 6361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414d66e80a1c4dfebecafc2af41c1d37, State: Initialized, Role: FOLLOWER
I20260924 16:04:44.208000 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:44.208021 6361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414d66e80a1c4dfebecafc2af41c1d37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37207 } }
I20260924 16:04:44.208282 6361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:44.208410 6361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:44.208554 6361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:44.211930 6361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414d66e80a1c4dfebecafc2af41c1d37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37207 } }
I20260924 16:04:44.212548 6361 leader_election.cc:302] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 414d66e80a1c4dfebecafc2af41c1d37; no voters:
I20260924 16:04:44.213315 6361 leader_election.cc:288] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:44.213441 6370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:44.214910 6370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [term 1 LEADER]: Becoming Leader. State: Replica: 414d66e80a1c4dfebecafc2af41c1d37, State: Running, Role: LEADER
I20260924 16:04:44.215643 6370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414d66e80a1c4dfebecafc2af41c1d37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37207 } }
I20260924 16:04:44.215822 6361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:44.218725 6374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 414d66e80a1c4dfebecafc2af41c1d37. Latest consensus state: current_term: 1 leader_uuid: "414d66e80a1c4dfebecafc2af41c1d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414d66e80a1c4dfebecafc2af41c1d37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37207 } } }
I20260924 16:04:44.219138 6374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:44.219702 6372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "414d66e80a1c4dfebecafc2af41c1d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414d66e80a1c4dfebecafc2af41c1d37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37207 } } }
I20260924 16:04:44.219990 6372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:44.223995 6379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:44.230067 6379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:44.234380 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:44.213255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:36821 (local address 127.0.0.1:37207)
0924 16:04:44.213453 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:44.213460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:44.214090 (+ 630us) server_negotiation.cc:407] Connection header received
0924 16:04:44.214508 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:44.214518 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:44.214628 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:44.214723 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:44.216988 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:44.218383 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:44.220387 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:44.222043 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:44.222266 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:44.225882 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:44.225900 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:44.225909 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:44.225956 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:44.229905 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:44.231031 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:44.231037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:44.231041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:44.231187 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:44.231818 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:44.231824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:44.231830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:44.232263 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:44.232509 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:44.233812 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0924 16:04:44.233992 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:04:44.241369 6379 catalog_manager.cc:1347] Generated new cluster ID: a194c760340f473c832033483f4e5c1d
I20260924 16:04:44.241501 6379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:44.267746 6379 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:44.269796 6379 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:44.284127 6379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 414d66e80a1c4dfebecafc2af41c1d37: Generated new TSK 0
I20260924 16:04:44.285071 6379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:44.296648 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6294
I20260924 16:04:44.305531 6356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:44.399973 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 6294
W20260924 16:04:44.417855 6278 connection.cc:535] client connection to 127.0.0.1:37207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37207): Connection reset by peer (error 104)
2026-09-24T16:04:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:44 dist-test-slave-xn4h krb5kdc[6395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:44 dist-test-slave-xn4h krb5kdc[6395](info): set up 2 sockets
Sep 24 16:04:44 dist-test-slave-xn4h krb5kdc[6395](info): commencing operation
krb5kdc: starting...
W20260924 16:04:45.524462 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:45 dist-test-slave-xn4h krb5kdc[6395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:45.814249 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:46.169255 6408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:46.169579 6408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:46.169672 6408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:46.177066 6408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:46.177204 6408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:46.177253 6408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:46.177309 6408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:46.177345 6408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:46.187669 6408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:46.189083 6408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:46.190538 6408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:46.203169 6413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:46.203145 6416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:46.203083 6414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:46.204221 6408 server_base.cc:1034] running on GCE node
I20260924 16:04:46.205055 6408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:46.206660 6408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:46.207872 6408 hybrid_clock.cc:648] HybridClock initialized: now 1790265886207832 us; error 27 us; skew 500 ppm
I20260924 16:04:46.208252 6408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:46 dist-test-slave-xn4h krb5kdc[6395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:46.211556 6408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:46.213153 6408 webserver.cc:460] Webserver started at http://127.0.0.1:42511/ using document root <none> and password file <none>
I20260924 16:04:46.213827 6408 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:46.213933 6408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:46.214193 6408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:46.217381 6408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2"
format_stamp: "Formatted at 2026-09-24 16:04:46 on dist-test-slave-xn4h"
server_key: "b41c689f5466606a30a116382967b3af"
server_key_iv: "d25c3ed551223d4878b8590427992fd3"
server_key_version: "encryptionkey@0"
I20260924 16:04:46.218166 6408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2"
format_stamp: "Formatted at 2026-09-24 16:04:46 on dist-test-slave-xn4h"
server_key: "b41c689f5466606a30a116382967b3af"
server_key_iv: "d25c3ed551223d4878b8590427992fd3"
server_key_version: "encryptionkey@0"
I20260924 16:04:46.223829 6408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:04:46.227671 6423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:46.229174 6408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:04:46.229405 6408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2"
format_stamp: "Formatted at 2026-09-24 16:04:46 on dist-test-slave-xn4h"
server_key: "b41c689f5466606a30a116382967b3af"
server_key_iv: "d25c3ed551223d4878b8590427992fd3"
server_key_version: "encryptionkey@0"
I20260924 16:04:46.229667 6408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:46.243860 6408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:46.248157 6408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:46.248477 6408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:46.268373 6408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45831
I20260924 16:04:46.268411 6475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45831 every 8 connection(s)
I20260924 16:04:46.269902 6408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:46.271184 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 6408
I20260924 16:04:46.271386 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:46.271939 17318 external_mini_cluster.cc:1428] Setting key 9e3642b57e4c4a401a8b3c12034d9985
I20260924 16:04:46.275100 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:46.285626 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:46.273930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:47663 (local address 127.0.0.1:45831)
0924 16:04:46.274790 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:46.274802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:46.274845 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:04:46.276550 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:46.276573 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:46.276982 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:46.277380 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:46.277993 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:46.279400 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:46.280804 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:46.281521 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:46.281727 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:46.283141 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:46.283149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:46.283165 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:46.283770 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:46.283857 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:46.283887 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:46.284084 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:46.284386 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 16:04:46.284595 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":420,"threads_started":1}
I20260924 16:04:46.286978 6476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2: Bootstrap starting.
I20260924 16:04:46.290004 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:46.290684 6476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:46.291743 6476 log.cc:824] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:46.295557 6476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2: No bootstrap required, opened a new log
I20260924 16:04:46.302222 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:46.294889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:57785 (local address 127.0.0.1:45831)
0924 16:04:46.295133 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:46.295142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:46.296055 (+ 913us) server_negotiation.cc:407] Connection header received
0924 16:04:46.296387 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:46.296395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:46.296509 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:46.296630 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:46.297191 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:46.298035 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:46.299524 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:46.299990 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:46.300151 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:46.301412 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:46.301417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:46.301420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:46.301440 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:46.301483 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:46.301499 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:46.301638 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:46.301833 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 16:04:46.301894 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:04:46.304564 6476 raft_consensus.cc:348] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } }
I20260924 16:04:46.305038 6476 raft_consensus.cc:374] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:46.305154 6476 raft_consensus.cc:729] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12d1da7c3aa46bf8473c4eaf4c27ee2, State: Initialized, Role: FOLLOWER
I20260924 16:04:46.305838 6476 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } }
I20260924 16:04:46.306103 6476 raft_consensus.cc:388] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:46.306257 6476 raft_consensus.cc:482] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:46.306515 6476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:46.309022 6476 raft_consensus.cc:504] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } }
I20260924 16:04:46.309644 6476 leader_election.cc:302] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12d1da7c3aa46bf8473c4eaf4c27ee2; no voters:
I20260924 16:04:46.310442 6476 leader_election.cc:288] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:46.310571 6486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:46.311229 6486 raft_consensus.cc:686] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [term 1 LEADER]: Becoming Leader. State: Replica: a12d1da7c3aa46bf8473c4eaf4c27ee2, State: Running, Role: LEADER
I20260924 16:04:46.311894 6486 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } }
I20260924 16:04:46.312804 6476 sys_catalog.cc:564] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:46.314738 6488 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12d1da7c3aa46bf8473c4eaf4c27ee2. Latest consensus state: current_term: 1 leader_uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } } }
I20260924 16:04:46.315135 6488 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:46.318063 6492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:46.317876 6487 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12d1da7c3aa46bf8473c4eaf4c27ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } } }
I20260924 16:04:46.318215 6487 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:46.323300 6492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:46.332536 6492 catalog_manager.cc:1347] Generated new cluster ID: dc9f3a07eecb4d6790d7da52717878c9
I20260924 16:04:46.332674 6492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:46.360371 6492 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:46.361814 6492 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:46.372911 6492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a12d1da7c3aa46bf8473c4eaf4c27ee2: Generated new TSK 0
I20260924 16:04:46.374117 6492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:46.392274 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6408
W20260924 16:04:46.393756 6471 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.6:47663
W20260924 16:04:46.394754 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:04:46.394946 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 6408
W20260924 16:04:46.414788 6393 connection.cc:535] client connection to 127.0.0.1:45831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45831): Connection reset by peer (error 104)
2026-09-24T16:04:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:46 dist-test-slave-xn4h krb5kdc[6507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:46 dist-test-slave-xn4h krb5kdc[6507](info): set up 2 sockets
Sep 24 16:04:46 dist-test-slave-xn4h krb5kdc[6507](info): commencing operation
krb5kdc: starting...
W20260924 16:04:47.532341 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:47 dist-test-slave-xn4h krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:47.805344 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:48.158491 6520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:48.158815 6520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:48.158977 6520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:48.166514 6520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:48.166644 6520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:48.166735 6520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:48.166827 6520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:48.166922 6520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:48.177182 6520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:48.178592 6520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:48.179986 6520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:48.192255 6527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:48.192472 6529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:48.192266 6526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:48.193133 6520 server_base.cc:1034] running on GCE node
I20260924 16:04:48.194027 6520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:48.195554 6520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:48.196749 6520 hybrid_clock.cc:648] HybridClock initialized: now 1790265888196710 us; error 29 us; skew 500 ppm
I20260924 16:04:48.197141 6520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:48 dist-test-slave-xn4h krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:48.200423 6520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:48.201977 6520 webserver.cc:460] Webserver started at http://127.0.0.1:33391/ using document root <none> and password file <none>
I20260924 16:04:48.202785 6520 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:48.202931 6520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:48.203243 6520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:48.206486 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "e52776f88c844051bd6596075c91ff8f"
format_stamp: "Formatted at 2026-09-24 16:04:48 on dist-test-slave-xn4h"
server_key: "e80b7da8413028fba7df37665a40bdf5"
server_key_iv: "fb15d73af1f1d259ba204dc12a26e2f7"
server_key_version: "encryptionkey@0"
I20260924 16:04:48.207326 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "e52776f88c844051bd6596075c91ff8f"
format_stamp: "Formatted at 2026-09-24 16:04:48 on dist-test-slave-xn4h"
server_key: "e80b7da8413028fba7df37665a40bdf5"
server_key_iv: "fb15d73af1f1d259ba204dc12a26e2f7"
server_key_version: "encryptionkey@0"
I20260924 16:04:48.212764 6520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 16:04:48.216463 6536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:48.217962 6520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:04:48.218158 6520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "e52776f88c844051bd6596075c91ff8f"
format_stamp: "Formatted at 2026-09-24 16:04:48 on dist-test-slave-xn4h"
server_key: "e80b7da8413028fba7df37665a40bdf5"
server_key_iv: "fb15d73af1f1d259ba204dc12a26e2f7"
server_key_version: "encryptionkey@0"
I20260924 16:04:48.218456 6520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:48.224665 6520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:48.228999 6520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:48.229357 6520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:48.248448 6520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42949
I20260924 16:04:48.248485 6588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42949 every 8 connection(s)
I20260924 16:04:48.249945 6520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:48.251770 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 6520
I20260924 16:04:48.251964 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:48.252526 17318 external_mini_cluster.cc:1428] Setting key c22157826b1a02d18df51d4c706a97df
I20260924 16:04:48.254843 6589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:48.260892 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:48.254435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:50487 (local address 127.0.0.1:42949)
0924 16:04:48.255419 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:48.255430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:48.255473 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:04:48.257167 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:48.257188 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:48.257568 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:48.257961 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:48.259163 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:48.259169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:48.259183 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:48.259200 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:48.259285 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:48.259307 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:48.259501 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:48.259616 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 16:04:48.259893 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":452,"threads_started":1}
I20260924 16:04:48.264962 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:48.267967 6589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f: Bootstrap starting.
I20260924 16:04:48.271718 6589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:48.272233 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:48.269546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:59673 (local address 127.0.0.1:42949)
0924 16:04:48.269742 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:48.269754 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:48.270200 (+ 446us) server_negotiation.cc:407] Connection header received
0924 16:04:48.270366 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:48.270372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:48.270441 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:48.270556 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:48.271519 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:48.271527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:48.271535 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:48.271556 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:48.271617 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:48.271640 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:48.271778 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:48.271830 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 16:04:48.271907 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:04:48.273029 6589 log.cc:824] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:48.275961 6589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f: No bootstrap required, opened a new log
I20260924 16:04:48.283780 6589 raft_consensus.cc:348] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52776f88c844051bd6596075c91ff8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } }
I20260924 16:04:48.284215 6589 raft_consensus.cc:374] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:48.284325 6589 raft_consensus.cc:729] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52776f88c844051bd6596075c91ff8f, State: Initialized, Role: FOLLOWER
I20260924 16:04:48.284994 6589 consensus_queue.cc:260] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52776f88c844051bd6596075c91ff8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } }
I20260924 16:04:48.285250 6589 raft_consensus.cc:388] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:48.285422 6589 raft_consensus.cc:482] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:48.285601 6589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:48.288206 6589 raft_consensus.cc:504] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52776f88c844051bd6596075c91ff8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } }
I20260924 16:04:48.288738 6589 leader_election.cc:302] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e52776f88c844051bd6596075c91ff8f; no voters:
I20260924 16:04:48.289333 6589 leader_election.cc:288] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:48.289497 6599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:48.290194 6599 raft_consensus.cc:686] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [term 1 LEADER]: Becoming Leader. State: Replica: e52776f88c844051bd6596075c91ff8f, State: Running, Role: LEADER
I20260924 16:04:48.290807 6599 consensus_queue.cc:237] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52776f88c844051bd6596075c91ff8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } }
I20260924 16:04:48.291581 6589 sys_catalog.cc:564] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:48.294215 6601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e52776f88c844051bd6596075c91ff8f. Latest consensus state: current_term: 1 leader_uuid: "e52776f88c844051bd6596075c91ff8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52776f88c844051bd6596075c91ff8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } } }
I20260924 16:04:48.294658 6601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:48.295832 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e52776f88c844051bd6596075c91ff8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52776f88c844051bd6596075c91ff8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } } }
I20260924 16:04:48.296185 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:48.297523 6609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:48.303391 6609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:48.312949 6609 catalog_manager.cc:1347] Generated new cluster ID: 63c3658149e9473892194b40ec098de9
I20260924 16:04:48.313175 6609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:48.328735 6609 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:48.329272 6609 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:48.341624 6609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e52776f88c844051bd6596075c91ff8f: Generated new TSK 0
I20260924 16:04:48.342849 6609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:48.356098 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6520
W20260924 16:04:48.357692 6584 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.6:50487
W20260924 16:04:48.358572 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:04:48.358768 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 6520
2026-09-24T16:04:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:48 dist-test-slave-xn4h krb5kdc[6620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:48 dist-test-slave-xn4h krb5kdc[6620](info): set up 2 sockets
Sep 24 16:04:48 dist-test-slave-xn4h krb5kdc[6620](info): commencing operation
krb5kdc: starting...
W20260924 16:04:49.484431 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:49 dist-test-slave-xn4h krb5kdc[6620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:49.757526 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:50.109143 6634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:50.109481 6634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:50.109579 6634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:50.117587 6634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:50.117782 6634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:50.117869 6634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:50.117959 6634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:50.118059 6634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:50.131003 6634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:50.132501 6634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:50.133884 6634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:50.146375 6639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:50.146469 6642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:50.146353 6640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:50.147456 6634 server_base.cc:1034] running on GCE node
I20260924 16:04:50.148509 6634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:50.150085 6634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:50.151304 6634 hybrid_clock.cc:648] HybridClock initialized: now 1790265890151242 us; error 52 us; skew 500 ppm
I20260924 16:04:50.151711 6634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:50 dist-test-slave-xn4h krb5kdc[6620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:50.169157 6634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:50.171000 6634 webserver.cc:460] Webserver started at http://127.0.0.1:37363/ using document root <none> and password file <none>
I20260924 16:04:50.171754 6634 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:50.171908 6634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:50.172236 6634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:50.175501 6634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "22fdc04090434bcf993339f33883a48f"
format_stamp: "Formatted at 2026-09-24 16:04:50 on dist-test-slave-xn4h"
server_key: "203238086fdde88a610b8ae6d277b760"
server_key_iv: "b26c10b69bc30a4b9872b61dd612d0cf"
server_key_version: "encryptionkey@0"
I20260924 16:04:50.176342 6634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "22fdc04090434bcf993339f33883a48f"
format_stamp: "Formatted at 2026-09-24 16:04:50 on dist-test-slave-xn4h"
server_key: "203238086fdde88a610b8ae6d277b760"
server_key_iv: "b26c10b69bc30a4b9872b61dd612d0cf"
server_key_version: "encryptionkey@0"
I20260924 16:04:50.181864 6634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:04:50.185621 6649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:50.187137 6634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:04:50.187345 6634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "22fdc04090434bcf993339f33883a48f"
format_stamp: "Formatted at 2026-09-24 16:04:50 on dist-test-slave-xn4h"
server_key: "203238086fdde88a610b8ae6d277b760"
server_key_iv: "b26c10b69bc30a4b9872b61dd612d0cf"
server_key_version: "encryptionkey@0"
I20260924 16:04:50.187572 6634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:50.200709 6634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:50.204901 6634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:50.205240 6634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:50.224884 6634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37583
I20260924 16:04:50.224906 6703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37583 every 8 connection(s)
I20260924 16:04:50.226354 6634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:50.227819 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 6634
I20260924 16:04:50.228016 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:50.228456 17318 external_mini_cluster.cc:1428] Setting key 0a18122245f7c2a04b21a0ccf85d9d4a
I20260924 16:04:50.231033 6704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:04:50 dist-test-slave-xn4h krb5kdc[6620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:04:50.242708 6704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f: Bootstrap starting.
I20260924 16:04:50.246452 6704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:50.247593 6704 log.cc:824] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:50.250818 6704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f: No bootstrap required, opened a new log
I20260924 16:04:50.251775 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:50.230251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:56533 (local address 127.0.0.1:37583)
0924 16:04:50.231182 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:50.231200 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:50.231263 (+ 63us) server_negotiation.cc:407] Connection header received
0924 16:04:50.232964 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:50.232989 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:50.233410 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:50.233881 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:50.235458 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:50.236846 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:50.238195 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:50.238970 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:50.239181 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:50.243698 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:50.243707 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:50.243724 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:50.243771 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:50.247153 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:50.248139 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:50.248147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:50.248154 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:50.248354 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:50.248872 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:50.248877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:50.248879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:50.249444 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:50.249780 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:50.250366 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 16:04:50.250705 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":379,"threads_started":1}
I20260924 16:04:50.255806 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:50.259250 6704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fdc04090434bcf993339f33883a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37583 } }
I20260924 16:04:50.259671 6704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:50.259822 6704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fdc04090434bcf993339f33883a48f, State: Initialized, Role: FOLLOWER
I20260924 16:04:50.260649 6704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fdc04090434bcf993339f33883a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37583 } }
I20260924 16:04:50.260967 6704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:50.261148 6704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:50.261353 6704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:50.264209 6704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fdc04090434bcf993339f33883a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37583 } }
I20260924 16:04:50.264786 6704 leader_election.cc:302] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22fdc04090434bcf993339f33883a48f; no voters:
I20260924 16:04:50.265403 6704 leader_election.cc:288] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:50.265561 6714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:50.266824 6714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [term 1 LEADER]: Becoming Leader. State: Replica: 22fdc04090434bcf993339f33883a48f, State: Running, Role: LEADER
I20260924 16:04:50.267511 6714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fdc04090434bcf993339f33883a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37583 } }
I20260924 16:04:50.267872 6704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:50.270255 6715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22fdc04090434bcf993339f33883a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fdc04090434bcf993339f33883a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37583 } } }
I20260924 16:04:50.270668 6715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:50.274547 6716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22fdc04090434bcf993339f33883a48f. Latest consensus state: current_term: 1 leader_uuid: "22fdc04090434bcf993339f33883a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fdc04090434bcf993339f33883a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37583 } } }
I20260924 16:04:50.274961 6716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:50.278911 6719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:50.281207 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:50.260283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:53677 (local address 127.0.0.1:37583)
0924 16:04:50.260460 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:50.260467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:50.260952 (+ 485us) server_negotiation.cc:407] Connection header received
0924 16:04:50.261167 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:50.261173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:50.261264 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:50.261403 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:50.263112 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:50.264258 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:50.265941 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:50.266963 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:50.267073 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:50.270908 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:50.270919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:50.270927 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:50.270983 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:50.276010 (+ 5027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:50.277496 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:50.277505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:50.277512 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:50.277777 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:50.278534 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:50.278539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:50.278542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:50.278998 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:50.280326 (+ 1328us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:50.280540 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 16:04:50.280739 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 16:04:50.284695 6719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:50.294081 6719 catalog_manager.cc:1347] Generated new cluster ID: ef8956f090844f81a8ad1bf7ce514e7a
I20260924 16:04:50.294219 6719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:50.307838 6719 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:50.309804 6719 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:50.330478 6719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22fdc04090434bcf993339f33883a48f: Generated new TSK 0
I20260924 16:04:50.331712 6719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:50.344488 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6634
I20260924 16:04:50.346252 6699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:50.440397 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 6634
W20260924 16:04:50.458285 6618 connection.cc:535] client connection to 127.0.0.1:37583 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37583): Connection reset by peer (error 104)
2026-09-24T16:04:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2081 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:50 dist-test-slave-xn4h krb5kdc[6737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:50 dist-test-slave-xn4h krb5kdc[6737](info): set up 2 sockets
Sep 24 16:04:50 dist-test-slave-xn4h krb5kdc[6737](info): commencing operation
krb5kdc: starting...
W20260924 16:04:51.564201 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:51 dist-test-slave-xn4h krb5kdc[6737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:51.829557 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:52.184520 6750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:52.184836 6750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:52.184928 6750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:52.192395 6750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:52.192533 6750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:52.192580 6750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:52.192642 6750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:52.192690 6750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:52.203162 6750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:52.204659 6750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:52.206067 6750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:52.219285 6758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:52.219383 6755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:52.219405 6756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:52.220696 6750 server_base.cc:1034] running on GCE node
I20260924 16:04:52.221648 6750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:52.223289 6750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:52.224573 6750 hybrid_clock.cc:648] HybridClock initialized: now 1790265892224523 us; error 41 us; skew 500 ppm
I20260924 16:04:52.224990 6750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:52 dist-test-slave-xn4h krb5kdc[6737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:52.228345 6750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:52.229993 6750 webserver.cc:460] Webserver started at http://127.0.0.1:45835/ using document root <none> and password file <none>
I20260924 16:04:52.230791 6750 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:52.230959 6750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:52.231289 6750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:52.234575 6750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "13acc6da635940508cac89b6b833b417"
format_stamp: "Formatted at 2026-09-24 16:04:52 on dist-test-slave-xn4h"
server_key: "c31268e00028eab5a9bdfcb526aa921a"
server_key_iv: "c7a756787960599f88ea9ac1ae60b540"
server_key_version: "encryptionkey@0"
I20260924 16:04:52.235430 6750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "13acc6da635940508cac89b6b833b417"
format_stamp: "Formatted at 2026-09-24 16:04:52 on dist-test-slave-xn4h"
server_key: "c31268e00028eab5a9bdfcb526aa921a"
server_key_iv: "c7a756787960599f88ea9ac1ae60b540"
server_key_version: "encryptionkey@0"
I20260924 16:04:52.241050 6750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:04:52.244850 6765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:52.246512 6750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:04:52.246661 6750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "13acc6da635940508cac89b6b833b417"
format_stamp: "Formatted at 2026-09-24 16:04:52 on dist-test-slave-xn4h"
server_key: "c31268e00028eab5a9bdfcb526aa921a"
server_key_iv: "c7a756787960599f88ea9ac1ae60b540"
server_key_version: "encryptionkey@0"
I20260924 16:04:52.246870 6750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:52.274572 6750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:52.279093 6750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:52.279345 6750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:52.298864 6750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36813
I20260924 16:04:52.298903 6818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36813 every 8 connection(s)
I20260924 16:04:52.300381 6750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:52.304272 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:52.306529 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 6750
I20260924 16:04:52.306725 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:52.307209 17318 external_mini_cluster.cc:1428] Setting key e93842ca2a02c09f8397d69f0c80b830
I20260924 16:04:52.317072 6819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417: Bootstrap starting.
I20260924 16:04:52.320964 6819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:52.321678 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:52.309443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:34385 (local address 127.0.0.1:36813)
0924 16:04:52.310459 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:52.310472 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:52.310515 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:04:52.312213 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:52.312238 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:52.312644 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:52.312990 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:52.313634 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:52.315087 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:52.316955 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:52.317709 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:52.317870 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:52.319208 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:52.319216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:52.319229 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:52.319765 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:52.319883 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:52.319917 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:52.320043 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:52.320334 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 16:04:52.320544 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":487,"threads_started":1}
I20260924 16:04:52.322268 6819 log.cc:824] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:52.325636 6819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417: No bootstrap required, opened a new log
I20260924 16:04:52.326048 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:52.333725 6819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13acc6da635940508cac89b6b833b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36813 } }
I20260924 16:04:52.334183 6819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:52.334295 6819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13acc6da635940508cac89b6b833b417, State: Initialized, Role: FOLLOWER
I20260924 16:04:52.335038 6819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13acc6da635940508cac89b6b833b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36813 } }
I20260924 16:04:52.335330 6819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:52.335506 6819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:52.335692 6819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:52.337397 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:52.330672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:59961 (local address 127.0.0.1:36813)
0924 16:04:52.330938 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:52.330947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:52.331467 (+ 520us) server_negotiation.cc:407] Connection header received
0924 16:04:52.331700 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:52.331704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:52.331795 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:52.331912 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:52.332436 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:52.333204 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:52.334730 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:52.335196 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:52.335364 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:52.336600 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:52.336605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:52.336608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:52.336628 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:52.336690 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:52.336707 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:52.336847 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:52.337053 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 16:04:52.337114 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:04:52.338425 6819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13acc6da635940508cac89b6b833b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36813 } }
I20260924 16:04:52.339069 6819 leader_election.cc:302] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13acc6da635940508cac89b6b833b417; no voters:
I20260924 16:04:52.339674 6819 leader_election.cc:288] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:52.339833 6829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:52.341176 6829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [term 1 LEADER]: Becoming Leader. State: Replica: 13acc6da635940508cac89b6b833b417, State: Running, Role: LEADER
I20260924 16:04:52.341763 6829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13acc6da635940508cac89b6b833b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36813 } }
I20260924 16:04:52.342202 6819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:52.345002 6830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13acc6da635940508cac89b6b833b417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13acc6da635940508cac89b6b833b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36813 } } }
I20260924 16:04:52.345027 6831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13acc6da635940508cac89b6b833b417. Latest consensus state: current_term: 1 leader_uuid: "13acc6da635940508cac89b6b833b417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13acc6da635940508cac89b6b833b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36813 } } }
I20260924 16:04:52.345484 6830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:52.345539 6831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:52.348521 6837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:52.354210 6837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:52.364176 6837 catalog_manager.cc:1347] Generated new cluster ID: 5667a8c9bc7347788974d15d41205c71
I20260924 16:04:52.364320 6837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:52.386166 6837 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:52.387584 6837 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:52.395764 6837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13acc6da635940508cac89b6b833b417: Generated new TSK 0
I20260924 16:04:52.396911 6837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:52.411856 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6750
W20260924 16:04:52.413157 6814 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.6:34385
W20260924 16:04:52.413954 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6750: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:04:52.414119 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 6750
W20260924 16:04:52.432016 6735 connection.cc:535] client connection to 127.0.0.1:36813 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36813): Connection reset by peer (error 104)
2026-09-24T16:04:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:52 dist-test-slave-xn4h krb5kdc[6851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:52 dist-test-slave-xn4h krb5kdc[6851](info): set up 2 sockets
Sep 24 16:04:52 dist-test-slave-xn4h krb5kdc[6851](info): commencing operation
krb5kdc: starting...
W20260924 16:04:53.528270 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:53 dist-test-slave-xn4h krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:04:53.801875 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:54.157341 6864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:54.157742 6864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:54.157917 6864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:54.165508 6864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:54.165694 6864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:54.165808 6864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:54.165927 6864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:54.166005 6864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:54.176584 6864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:54.178061 6864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:54.179502 6864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:54.192287 6873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:54.192338 6870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:54.192319 6871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:54.193148 6864 server_base.cc:1034] running on GCE node
I20260924 16:04:54.194046 6864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:54.195627 6864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:54.196862 6864 hybrid_clock.cc:648] HybridClock initialized: now 1790265894196830 us; error 36 us; skew 500 ppm
I20260924 16:04:54.197275 6864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:54 dist-test-slave-xn4h krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:54.200585 6864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:04:54.202206 6864 webserver.cc:460] Webserver started at http://127.0.0.1:34735/ using document root <none> and password file <none>
I20260924 16:04:54.202961 6864 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:54.203115 6864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:54.203450 6864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:54.207206 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "674fa203530348559721f8e82ed04203"
format_stamp: "Formatted at 2026-09-24 16:04:54 on dist-test-slave-xn4h"
server_key: "7accce832b1acd818754d7ee72d5ceb6"
server_key_iv: "8e8ca1640143c54f0391c857c0495ea7"
server_key_version: "encryptionkey@0"
I20260924 16:04:54.208055 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "674fa203530348559721f8e82ed04203"
format_stamp: "Formatted at 2026-09-24 16:04:54 on dist-test-slave-xn4h"
server_key: "7accce832b1acd818754d7ee72d5ceb6"
server_key_iv: "8e8ca1640143c54f0391c857c0495ea7"
server_key_version: "encryptionkey@0"
I20260924 16:04:54.213413 6864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:04:54.217119 6880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:54.218667 6864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:04:54.218883 6864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "674fa203530348559721f8e82ed04203"
format_stamp: "Formatted at 2026-09-24 16:04:54 on dist-test-slave-xn4h"
server_key: "7accce832b1acd818754d7ee72d5ceb6"
server_key_iv: "8e8ca1640143c54f0391c857c0495ea7"
server_key_version: "encryptionkey@0"
I20260924 16:04:54.219123 6864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:54.225322 6864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:54.229756 6864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:54.230079 6864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:54.249662 6864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36479
I20260924 16:04:54.249713 6932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36479 every 8 connection(s)
I20260924 16:04:54.251214 6864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:54.255146 6933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:54.258471 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 6864
I20260924 16:04:54.258661 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:54.259125 17318 external_mini_cluster.cc:1428] Setting key 50e6e4a90130e7abad7efdc458ffe49c
I20260924 16:04:54.267365 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:54.261065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:60411 (local address 127.0.0.1:36479)
0924 16:04:54.261903 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:54.261915 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:54.261961 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:04:54.263736 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:54.263760 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:54.264160 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:54.264524 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:54.265689 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:54.265697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:54.265711 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:54.265727 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:54.265789 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:54.265808 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:54.266024 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:54.266123 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 16:04:54.266382 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":389,"threads_started":1}
I20260924 16:04:54.268194 6933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203: Bootstrap starting.
I20260924 16:04:54.271579 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:54.272127 6933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:54.273496 6933 log.cc:824] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:54.276851 6933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203: No bootstrap required, opened a new log
I20260924 16:04:54.279528 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:54.276885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:54321 (local address 127.0.0.1:36479)
0924 16:04:54.277082 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:54.277087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:54.277444 (+ 357us) server_negotiation.cc:407] Connection header received
0924 16:04:54.277610 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:54.277616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:54.277680 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:54.277806 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:54.278790 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:54.278796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:54.278802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:54.278828 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:54.278882 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:54.278903 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:54.279040 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:54.279086 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 16:04:54.279169 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:04:54.285789 6933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674fa203530348559721f8e82ed04203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36479 } }
I20260924 16:04:54.286191 6933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:54.286266 6933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674fa203530348559721f8e82ed04203, State: Initialized, Role: FOLLOWER
I20260924 16:04:54.286867 6933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674fa203530348559721f8e82ed04203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36479 } }
I20260924 16:04:54.287091 6933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:54.287194 6933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:54.287320 6933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:54.289691 6933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674fa203530348559721f8e82ed04203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36479 } }
I20260924 16:04:54.290174 6933 leader_election.cc:302] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 674fa203530348559721f8e82ed04203; no voters:
I20260924 16:04:54.290789 6933 leader_election.cc:288] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:54.290964 6944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:54.291627 6944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [term 1 LEADER]: Becoming Leader. State: Replica: 674fa203530348559721f8e82ed04203, State: Running, Role: LEADER
I20260924 16:04:54.292157 6944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674fa203530348559721f8e82ed04203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36479 } }
I20260924 16:04:54.292927 6933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:54.295436 6946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674fa203530348559721f8e82ed04203. Latest consensus state: current_term: 1 leader_uuid: "674fa203530348559721f8e82ed04203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674fa203530348559721f8e82ed04203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36479 } } }
I20260924 16:04:54.295915 6946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:54.296798 6945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "674fa203530348559721f8e82ed04203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674fa203530348559721f8e82ed04203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36479 } } }
I20260924 16:04:54.297215 6945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:54.298830 6953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:54.304386 6953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:54.314206 6953 catalog_manager.cc:1347] Generated new cluster ID: 2489d69119544b2092d24a9ed7a36515
I20260924 16:04:54.314414 6953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:54.331106 6953 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:54.331650 6953 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:54.338402 6953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 674fa203530348559721f8e82ed04203: Generated new TSK 0
I20260924 16:04:54.339285 6953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:54.346592 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6864
W20260924 16:04:54.347791 6928 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.6:60411
W20260924 16:04:54.348584 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:04:54.348747 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 6864
2026-09-24T16:04:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:54 dist-test-slave-xn4h krb5kdc[6966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:54 dist-test-slave-xn4h krb5kdc[6966](info): set up 2 sockets
Sep 24 16:04:54 dist-test-slave-xn4h krb5kdc[6966](info): commencing operation
krb5kdc: starting...
W20260924 16:04:55.460047 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:55 dist-test-slave-xn4h krb5kdc[6966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:04:55.729746 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34053
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:56.094743 6979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:56.095124 6979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:56.095293 6979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:56.103252 6979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:56.103469 6979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:56.103585 6979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:56.103701 6979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:56.103788 6979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:56.114187 6979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:34053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:56.115703 6979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:56.117537 6979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:56.132575 6984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:56.132783 6987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:56.132592 6985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:56.133606 6979 server_base.cc:1034] running on GCE node
I20260924 16:04:56.134519 6979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:56.136128 6979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:56.137341 6979 hybrid_clock.cc:648] HybridClock initialized: now 1790265896137296 us; error 46 us; skew 500 ppm
I20260924 16:04:56.137742 6979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:56 dist-test-slave-xn4h krb5kdc[6966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265896, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:56.141136 6979 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:04:56.142849 6979 webserver.cc:460] Webserver started at http://127.16.233.190:37201/ using document root <none> and password file <none>
I20260924 16:04:56.143601 6979 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:56.143759 6979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:56.144085 6979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:56.147398 6979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "31e2467abd8d4abd84c311abd58f4bad"
format_stamp: "Formatted at 2026-09-24 16:04:56 on dist-test-slave-xn4h"
server_key: "08560d4ff63caef16bd998bb7d61b246"
server_key_iv: "6063a0ea9facba3f77267bfebf112fd8"
server_key_version: "encryptionkey@0"
I20260924 16:04:56.148313 6979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "31e2467abd8d4abd84c311abd58f4bad"
format_stamp: "Formatted at 2026-09-24 16:04:56 on dist-test-slave-xn4h"
server_key: "08560d4ff63caef16bd998bb7d61b246"
server_key_iv: "6063a0ea9facba3f77267bfebf112fd8"
server_key_version: "encryptionkey@0"
I20260924 16:04:56.154019 6979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 16:04:56.157768 6994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:56.159366 6979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:56.159595 6979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "31e2467abd8d4abd84c311abd58f4bad"
format_stamp: "Formatted at 2026-09-24 16:04:56 on dist-test-slave-xn4h"
server_key: "08560d4ff63caef16bd998bb7d61b246"
server_key_iv: "6063a0ea9facba3f77267bfebf112fd8"
server_key_version: "encryptionkey@0"
I20260924 16:04:56.159860 6979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:56.181802 6979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:56.186370 6979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:56.186689 6979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:56.206277 6979 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:34053
I20260924 16:04:56.206352 7046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:34053 every 8 connection(s)
I20260924 16:04:56.207885 6979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:56.209829 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 6979
I20260924 16:04:56.210035 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:56.210704 17318 external_mini_cluster.cc:1428] Setting key 227c2765dc1684db41f3b291574b986c
I20260924 16:04:56.212752 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:04:56 dist-test-slave-xn4h krb5kdc[6966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:04:56.227068 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad: Bootstrap starting.
I20260924 16:04:56.230424 7047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:56.231500 7047 log.cc:824] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:56.234634 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad: No bootstrap required, opened a new log
I20260924 16:04:56.235165 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:56.212561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48369 (local address 127.16.233.190:34053)
0924 16:04:56.213429 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:56.213441 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:56.213484 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:04:56.215260 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:56.215285 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:56.215718 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:56.216085 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:56.217966 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:56.219459 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:56.221278 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:56.221991 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:56.222126 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:56.227342 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:56.227352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:56.227369 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:56.227410 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:56.230693 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:56.231587 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:56.231594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:56.231601 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:56.231812 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:56.232308 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:56.232313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:56.232317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:56.232865 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:56.233230 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:56.233733 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 16:04:56.234056 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":441,"threads_started":1}
I20260924 16:04:56.239183 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:56.242894 7047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e2467abd8d4abd84c311abd58f4bad" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34053 } }
I20260924 16:04:56.243340 7047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:56.243748 7047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e2467abd8d4abd84c311abd58f4bad, State: Initialized, Role: FOLLOWER
I20260924 16:04:56.244611 7047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e2467abd8d4abd84c311abd58f4bad" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34053 } }
I20260924 16:04:56.244920 7047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:56.245092 7047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:56.245289 7047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:56.248390 7047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e2467abd8d4abd84c311abd58f4bad" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34053 } }
I20260924 16:04:56.249047 7047 leader_election.cc:302] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31e2467abd8d4abd84c311abd58f4bad; no voters:
I20260924 16:04:56.249725 7047 leader_election.cc:288] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:56.249889 7057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:56.251180 7057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [term 1 LEADER]: Becoming Leader. State: Replica: 31e2467abd8d4abd84c311abd58f4bad, State: Running, Role: LEADER
I20260924 16:04:56.251899 7057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e2467abd8d4abd84c311abd58f4bad" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34053 } }
I20260924 16:04:56.252084 7047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:56.262212 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31e2467abd8d4abd84c311abd58f4bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e2467abd8d4abd84c311abd58f4bad" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34053 } } }
I20260924 16:04:56.262763 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:56.262858 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31e2467abd8d4abd84c311abd58f4bad. Latest consensus state: current_term: 1 leader_uuid: "31e2467abd8d4abd84c311abd58f4bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e2467abd8d4abd84c311abd58f4bad" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 34053 } } }
I20260924 16:04:56.264043 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:56.265679 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:56.243660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46937 (local address 127.16.233.190:34053)
0924 16:04:56.243913 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:56.243923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:56.244344 (+ 421us) server_negotiation.cc:407] Connection header received
0924 16:04:56.244556 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:56.244559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:56.244626 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:56.244703 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:56.246687 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:56.247485 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:56.249222 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:56.249944 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:56.250089 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:56.254018 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:56.254029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:56.254037 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:56.254104 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:56.259115 (+ 5011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:56.260473 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:56.260478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:56.260482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:56.260661 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:56.261339 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:56.261344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:56.261347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:56.261734 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:56.263124 (+ 1390us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:56.263377 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 16:04:56.263950 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:04:56.271566 7065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:56.277446 7065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:56.288947 7065 catalog_manager.cc:1347] Generated new cluster ID: 50f035621f1542da9b2fac906bb3f00f
I20260924 16:04:56.289098 7065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:56.303771 7065 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:56.305745 7065 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:56.317927 7065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31e2467abd8d4abd84c311abd58f4bad: Generated new TSK 0
I20260924 16:04:56.319175 7065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:56.325153 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 6979
I20260924 16:04:56.327358 7042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:04:56.419318 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 6979
W20260924 16:04:56.437933 6964 connection.cc:535] client connection to 127.16.233.190:34053 recv error: Network error: failed to read from TLS socket (remote: 127.16.233.190:34053): Connection reset by peer (error 104)
2026-09-24T16:04:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2074 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:56 dist-test-slave-xn4h krb5kdc[7080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:56 dist-test-slave-xn4h krb5kdc[7080](info): set up 2 sockets
Sep 24 16:04:56 dist-test-slave-xn4h krb5kdc[7080](info): commencing operation
krb5kdc: starting...
W20260924 16:04:57.540278 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:57 dist-test-slave-xn4h krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:04:57.818738 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:40113
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:40113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:58.163653 7094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:58.163937 7094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:58.164088 7094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:58.171535 7094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:58.171684 7094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:58.171790 7094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:58.171885 7094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:58.171967 7094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:58.182564 7094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:40113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:04:58.183965 7094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:58.185384 7094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:58.197225 7100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:58.197288 7099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:58.197299 7102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:58.198272 7094 server_base.cc:1034] running on GCE node
I20260924 16:04:58.199113 7094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:58.200578 7094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:58.201799 7094 hybrid_clock.cc:648] HybridClock initialized: now 1790265898201744 us; error 50 us; skew 500 ppm
I20260924 16:04:58.202167 7094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:04:58 dist-test-slave-xn4h krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:58.205389 7094 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:04:58.207003 7094 webserver.cc:460] Webserver started at http://127.16.233.190:45161/ using document root <none> and password file <none>
I20260924 16:04:58.207729 7094 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:58.207875 7094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:58.208194 7094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:58.211320 7094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "67260e8c362d4e64a69e3806eb5e2645"
format_stamp: "Formatted at 2026-09-24 16:04:58 on dist-test-slave-xn4h"
server_key: "d0edd173872eedd3d78b90a8b7fe5841"
server_key_iv: "9c7339401c3b2c4465c4f054cad4969f"
server_key_version: "encryptionkey@0"
I20260924 16:04:58.212083 7094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "67260e8c362d4e64a69e3806eb5e2645"
format_stamp: "Formatted at 2026-09-24 16:04:58 on dist-test-slave-xn4h"
server_key: "d0edd173872eedd3d78b90a8b7fe5841"
server_key_iv: "9c7339401c3b2c4465c4f054cad4969f"
server_key_version: "encryptionkey@0"
I20260924 16:04:58.217154 7094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:04:58.220530 7109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:58.221822 7094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:04:58.221999 7094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "67260e8c362d4e64a69e3806eb5e2645"
format_stamp: "Formatted at 2026-09-24 16:04:58 on dist-test-slave-xn4h"
server_key: "d0edd173872eedd3d78b90a8b7fe5841"
server_key_iv: "9c7339401c3b2c4465c4f054cad4969f"
server_key_version: "encryptionkey@0"
I20260924 16:04:58.222191 7094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:58.237653 7094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:58.241595 7094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:58.241884 7094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:58.260047 7094 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:40113
I20260924 16:04:58.260076 7161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:40113 every 8 connection(s)
I20260924 16:04:58.261474 7094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:58.263851 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 7094
I20260924 16:04:58.264057 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:04:58.264513 17318 external_mini_cluster.cc:1428] Setting key fac7fb59ad04c7f9fda1ba829dd4726b
I20260924 16:04:58.265596 7162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:58.277593 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:58.266214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40937 (local address 127.16.233.190:40113)
0924 16:04:58.267117 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:58.267135 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:58.267428 (+ 293us) server_negotiation.cc:407] Connection header received
0924 16:04:58.269170 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:58.269191 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:58.269578 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:58.269973 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:58.270551 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:58.271861 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:58.273208 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:58.273850 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:58.274058 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:58.275378 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:58.275386 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:58.275400 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:58.275953 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:58.276042 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:58.276070 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:58.276243 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:58.276479 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 16:04:58.276649 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":374,"threads_started":1}
I20260924 16:04:58.278049 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645: Bootstrap starting.
I20260924 16:04:58.281503 7162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:58.281554 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:58.282600 7162 log.cc:824] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:58.285567 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645: No bootstrap required, opened a new log
I20260924 16:04:58.292842 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:58.285892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55343 (local address 127.16.233.190:40113)
0924 16:04:58.286109 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:58.286118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:58.286637 (+ 519us) server_negotiation.cc:407] Connection header received
0924 16:04:58.286819 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:58.286822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:58.286878 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:58.286959 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:58.287547 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:58.288337 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:58.290201 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:58.290725 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:58.290820 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:58.292066 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:58.292070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:58.292074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:04:58.292092 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:58.292130 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:04:58.292144 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:58.292232 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:58.292434 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 16:04:58.292478 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:04:58.295408 7162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67260e8c362d4e64a69e3806eb5e2645" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40113 } }
I20260924 16:04:58.295835 7162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:58.295950 7162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67260e8c362d4e64a69e3806eb5e2645, State: Initialized, Role: FOLLOWER
I20260924 16:04:58.296587 7162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67260e8c362d4e64a69e3806eb5e2645" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40113 } }
I20260924 16:04:58.296837 7162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:58.296991 7162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:58.297179 7162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:58.299990 7162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67260e8c362d4e64a69e3806eb5e2645" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40113 } }
I20260924 16:04:58.300544 7162 leader_election.cc:302] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67260e8c362d4e64a69e3806eb5e2645; no voters:
I20260924 16:04:58.301119 7162 leader_election.cc:288] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:58.301254 7172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:58.301940 7172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [term 1 LEADER]: Becoming Leader. State: Replica: 67260e8c362d4e64a69e3806eb5e2645, State: Running, Role: LEADER
I20260924 16:04:58.302606 7172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67260e8c362d4e64a69e3806eb5e2645" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40113 } }
I20260924 16:04:58.303472 7162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:58.305769 7174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67260e8c362d4e64a69e3806eb5e2645. Latest consensus state: current_term: 1 leader_uuid: "67260e8c362d4e64a69e3806eb5e2645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67260e8c362d4e64a69e3806eb5e2645" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40113 } } }
I20260924 16:04:58.305811 7173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67260e8c362d4e64a69e3806eb5e2645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67260e8c362d4e64a69e3806eb5e2645" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 40113 } } }
I20260924 16:04:58.306160 7174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:58.306207 7173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:58.307286 7179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:58.314401 7179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:58.323616 7179 catalog_manager.cc:1347] Generated new cluster ID: f05b4cfeefcd4121a4e34737c76c4322
I20260924 16:04:58.323755 7179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:58.339010 7179 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:58.340860 7179 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:58.348978 7179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67260e8c362d4e64a69e3806eb5e2645: Generated new TSK 0
I20260924 16:04:58.349897 7179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:58.358139 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7094
W20260924 16:04:58.359357 7157 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40937
W20260924 16:04:58.360105 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7094: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:04:58.360258 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 7094
W20260924 16:04:58.378134 7078 connection.cc:535] client connection to 127.16.233.190:40113 recv error: Network error: failed to read from TLS socket (remote: 127.16.233.190:40113): Connection reset by peer (error 104)
2026-09-24T16:04:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:58 dist-test-slave-xn4h krb5kdc[7193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:58 dist-test-slave-xn4h krb5kdc[7193](info): set up 2 sockets
Sep 24 16:04:58 dist-test-slave-xn4h krb5kdc[7193](info): commencing operation
krb5kdc: starting...
W20260924 16:04:59.476198 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:04:59 dist-test-slave-xn4h krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:04:59.747614 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:43189
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:44035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:00.090782 7206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:00.091084 7206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:00.091238 7206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:00.098845 7206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:00.098991 7206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:00.099090 7206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:00.099195 7206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:00.099279 7206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:00.109519 7206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:44035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:43189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:00.110888 7206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:00.112210 7206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:00.123898 7211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:00.123920 7212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:00.125447 7214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:00.126468 7206 server_base.cc:1034] running on GCE node
I20260924 16:05:00.127619 7206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:00.129283 7206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:00.130530 7206 hybrid_clock.cc:648] HybridClock initialized: now 1790265900130465 us; error 58 us; skew 500 ppm
I20260924 16:05:00.130908 7206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:00 dist-test-slave-xn4h krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:00.134117 7206 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:05:00.135771 7206 webserver.cc:460] Webserver started at http://127.16.233.190:45619/ using document root <none> and password file <none>
I20260924 16:05:00.136538 7206 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:00.136696 7206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:00.137015 7206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:00.140177 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "c6fb626420b848faab8f81d2eb996052"
format_stamp: "Formatted at 2026-09-24 16:05:00 on dist-test-slave-xn4h"
server_key: "de6d0e3a956d3aed6b84cf0b7cc4eb0f"
server_key_iv: "2a9c8128ee86e41a771a9cf60d24bc80"
server_key_version: "encryptionkey@0"
I20260924 16:05:00.140961 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "c6fb626420b848faab8f81d2eb996052"
format_stamp: "Formatted at 2026-09-24 16:05:00 on dist-test-slave-xn4h"
server_key: "de6d0e3a956d3aed6b84cf0b7cc4eb0f"
server_key_iv: "2a9c8128ee86e41a771a9cf60d24bc80"
server_key_version: "encryptionkey@0"
I20260924 16:05:00.146404 7206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 16:05:00.149886 7221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:00.151355 7206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 16:05:00.151553 7206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "c6fb626420b848faab8f81d2eb996052"
format_stamp: "Formatted at 2026-09-24 16:05:00 on dist-test-slave-xn4h"
server_key: "de6d0e3a956d3aed6b84cf0b7cc4eb0f"
server_key_iv: "2a9c8128ee86e41a771a9cf60d24bc80"
server_key_version: "encryptionkey@0"
I20260924 16:05:00.151763 7206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:00.157724 7206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:00.162007 7206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:00.162421 7206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:00.181149 7206 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:43189
I20260924 16:05:00.181186 7273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:43189 every 8 connection(s)
I20260924 16:05:00.182659 7206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:00.183095 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 7206
I20260924 16:05:00.183297 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:05:00.183902 17318 external_mini_cluster.cc:1428] Setting key f4472410bf4710c741aee52156eec125
I20260924 16:05:00.187381 7274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:00.192030 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:00.185964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59651 (local address 127.16.233.190:43189)
0924 16:05:00.186863 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:00.186874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:00.186915 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:05:00.188632 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:00.188655 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:00.188989 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:00.189324 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:00.190452 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:00.190458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:00.190470 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:00.190486 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:00.190565 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:00.190588 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:00.190756 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:00.190865 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 16:05:00.191068 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":453,"threads_started":1}
I20260924 16:05:00.196117 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:05:00.200958 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052: Bootstrap starting.
I20260924 16:05:00.204324 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:00.201528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44339 (local address 127.16.233.190:43189)
0924 16:05:00.201713 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:00.201719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:00.202217 (+ 498us) server_negotiation.cc:407] Connection header received
0924 16:05:00.202445 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:00.202452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:00.202524 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:00.202657 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:00.203666 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:00.203671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:00.203676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:00.203693 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:00.203742 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:00.203764 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:00.203883 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:00.203966 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 16:05:00.204031 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:05:00.204967 7274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:00.206254 7274 log.cc:824] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:00.209106 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052: No bootstrap required, opened a new log
I20260924 16:05:00.216714 7274 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb626420b848faab8f81d2eb996052" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43189 } }
I20260924 16:05:00.217186 7274 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:00.217324 7274 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6fb626420b848faab8f81d2eb996052, State: Initialized, Role: FOLLOWER
I20260924 16:05:00.217952 7274 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb626420b848faab8f81d2eb996052" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43189 } }
I20260924 16:05:00.218194 7274 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:00.218350 7274 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:00.218541 7274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:00.220729 7274 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb626420b848faab8f81d2eb996052" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43189 } }
I20260924 16:05:00.221264 7274 leader_election.cc:302] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6fb626420b848faab8f81d2eb996052; no voters:
I20260924 16:05:00.222028 7274 leader_election.cc:288] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:00.222137 7284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:00.222841 7284 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [term 1 LEADER]: Becoming Leader. State: Replica: c6fb626420b848faab8f81d2eb996052, State: Running, Role: LEADER
I20260924 16:05:00.223394 7284 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb626420b848faab8f81d2eb996052" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43189 } }
I20260924 16:05:00.224378 7274 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:00.226375 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6fb626420b848faab8f81d2eb996052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb626420b848faab8f81d2eb996052" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43189 } } }
I20260924 16:05:00.226477 7286 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6fb626420b848faab8f81d2eb996052. Latest consensus state: current_term: 1 leader_uuid: "c6fb626420b848faab8f81d2eb996052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb626420b848faab8f81d2eb996052" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43189 } } }
I20260924 16:05:00.226859 7286 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:00.226873 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:00.232136 7289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:00.237277 7289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:00.245950 7289 catalog_manager.cc:1347] Generated new cluster ID: bba1eb1abfd646f6b8275379cc9d12af
I20260924 16:05:00.246076 7289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:00.264231 7289 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:00.264756 7289 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:00.272287 7289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6fb626420b848faab8f81d2eb996052: Generated new TSK 0
I20260924 16:05:00.273291 7289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:05:00.287426 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7206
W20260924 16:05:00.288489 7269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59651
W20260924 16:05:00.289192 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7206: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:00.289345 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 7206
2026-09-24T16:05:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:00 dist-test-slave-xn4h krb5kdc[7306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:00 dist-test-slave-xn4h krb5kdc[7306](info): set up 2 sockets
Sep 24 16:05:00 dist-test-slave-xn4h krb5kdc[7306](info): commencing operation
krb5kdc: starting...
W20260924 16:05:01.396239 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:05:01 dist-test-slave-xn4h krb5kdc[7306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:05:01.661667 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:39869
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:39869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:02.002660 7319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:02.003134 7319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:02.003304 7319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:02.011046 7319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:02.011209 7319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:02.011320 7319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:02.011416 7319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:02.011484 7319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:02.021862 7319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:39869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:39869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:02.023347 7319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:02.024768 7319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:02.037250 7324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:02.037287 7327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:02.037335 7325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:02.038236 7319 server_base.cc:1034] running on GCE node
I20260924 16:05:02.039316 7319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:02.040932 7319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:02.042187 7319 hybrid_clock.cc:648] HybridClock initialized: now 1790265902042142 us; error 56 us; skew 500 ppm
I20260924 16:05:02.042609 7319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:02 dist-test-slave-xn4h krb5kdc[7306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265902, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:02.050684 7319 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:05:02.052234 7319 webserver.cc:460] Webserver started at http://127.16.233.190:43591/ using document root <none> and password file <none>
I20260924 16:05:02.052963 7319 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:02.053076 7319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:02.053359 7319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:02.056690 7319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "6bb31ffdaaee4a83a8674e4430521a01"
format_stamp: "Formatted at 2026-09-24 16:05:02 on dist-test-slave-xn4h"
server_key: "4094d9769116385de0c3ffcbddf68aa6"
server_key_iv: "b5ad5b846eb48abc555ccf02afe6f088"
server_key_version: "encryptionkey@0"
I20260924 16:05:02.057471 7319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "6bb31ffdaaee4a83a8674e4430521a01"
format_stamp: "Formatted at 2026-09-24 16:05:02 on dist-test-slave-xn4h"
server_key: "4094d9769116385de0c3ffcbddf68aa6"
server_key_iv: "b5ad5b846eb48abc555ccf02afe6f088"
server_key_version: "encryptionkey@0"
I20260924 16:05:02.063210 7319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 16:05:02.066905 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:02.068325 7319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 16:05:02.068532 7319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "6bb31ffdaaee4a83a8674e4430521a01"
format_stamp: "Formatted at 2026-09-24 16:05:02 on dist-test-slave-xn4h"
server_key: "4094d9769116385de0c3ffcbddf68aa6"
server_key_iv: "b5ad5b846eb48abc555ccf02afe6f088"
server_key_version: "encryptionkey@0"
I20260924 16:05:02.068743 7319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:02.092653 7319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:02.098824 7319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:02.099171 7319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:02.123279 7319 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:39869
I20260924 16:05:02.123288 7385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:39869 every 8 connection(s)
I20260924 16:05:02.125335 7319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:02.128940 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 7319
I20260924 16:05:02.129141 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:05:02.129562 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:02.129616 17318 external_mini_cluster.cc:1428] Setting key 6abef35cbb3c1277cae9d5e1f7dca08c
I20260924 16:05:02.145326 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01: Bootstrap starting.
Sep 24 16:05:02 dist-test-slave-xn4h krb5kdc[7306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:05:02.149084 7386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:02.150254 7386 log.cc:824] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:02.153384 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01: No bootstrap required, opened a new log
I20260924 16:05:02.155047 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:02.131598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43233 (local address 127.16.233.190:39869)
0924 16:05:02.132439 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:02.132454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:02.132497 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:05:02.134594 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:02.134620 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:02.135094 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:02.135488 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:02.137451 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:02.138966 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:02.140866 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:02.141640 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:02.141796 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:02.146985 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:02.146995 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:02.147011 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:02.147058 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:02.150526 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:02.151455 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:02.151463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:02.151470 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:02.151650 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:02.152054 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:02.152060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:02.152063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:02.152617 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:02.152932 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:02.153417 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 16:05:02.153888 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":385,"threads_started":1}
I20260924 16:05:02.158986 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:05:02.161686 7386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb31ffdaaee4a83a8674e4430521a01" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39869 } }
I20260924 16:05:02.162087 7386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:02.162225 7386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb31ffdaaee4a83a8674e4430521a01, State: Initialized, Role: FOLLOWER
I20260924 16:05:02.162897 7386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb31ffdaaee4a83a8674e4430521a01" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39869 } }
I20260924 16:05:02.163159 7386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:02.163337 7386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:02.163528 7386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:02.166174 7386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb31ffdaaee4a83a8674e4430521a01" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39869 } }
I20260924 16:05:02.166788 7386 leader_election.cc:302] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bb31ffdaaee4a83a8674e4430521a01; no voters:
I20260924 16:05:02.167480 7386 leader_election.cc:288] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:02.167616 7397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:02.168874 7397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [term 1 LEADER]: Becoming Leader. State: Replica: 6bb31ffdaaee4a83a8674e4430521a01, State: Running, Role: LEADER
I20260924 16:05:02.169574 7397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb31ffdaaee4a83a8674e4430521a01" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39869 } }
I20260924 16:05:02.169847 7386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:02.172740 7399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bb31ffdaaee4a83a8674e4430521a01. Latest consensus state: current_term: 1 leader_uuid: "6bb31ffdaaee4a83a8674e4430521a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb31ffdaaee4a83a8674e4430521a01" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39869 } } }
I20260924 16:05:02.173177 7399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:02.173725 7398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bb31ffdaaee4a83a8674e4430521a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb31ffdaaee4a83a8674e4430521a01" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 39869 } } }
I20260924 16:05:02.174036 7398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:02.182729 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:02.163418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37159 (local address 127.16.233.190:39869)
0924 16:05:02.163621 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:02.163629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:02.164081 (+ 452us) server_negotiation.cc:407] Connection header received
0924 16:05:02.164214 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:02.164221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:02.164318 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:02.164430 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:02.165898 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:02.167016 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:02.169058 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:02.169780 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:02.169916 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:02.173641 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:02.173651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:02.173658 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:02.173707 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:02.178349 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:02.179554 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:02.179564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:02.179570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:02.179816 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:02.180419 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:02.180439 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:02.180445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:02.180956 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:02.181274 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:02.182010 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 16:05:02.182238 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:05:02.183856 7403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:02.190866 7403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:02.203405 7403 catalog_manager.cc:1347] Generated new cluster ID: 8b96d2e80f984000833b20149efee4cd
I20260924 16:05:02.203540 7403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:02.232458 7403 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:02.233939 7403 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:02.244891 7403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bb31ffdaaee4a83a8674e4430521a01: Generated new TSK 0
I20260924 16:05:02.246037 7403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:05:02.263911 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7319
I20260924 16:05:02.265163 7381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:02.360986 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 7319
2026-09-24T16:05:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2071 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:02 dist-test-slave-xn4h krb5kdc[7420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:02 dist-test-slave-xn4h krb5kdc[7420](info): set up 2 sockets
Sep 24 16:05:02 dist-test-slave-xn4h krb5kdc[7420](info): commencing operation
krb5kdc: starting...
W20260924 16:05:03.472154 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:05:03 dist-test-slave-xn4h krb5kdc[7420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:05:03.738154 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:43009
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:35281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:43009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:04.080258 7433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:04.080574 7433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:04.080668 7433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:04.088325 7433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:04.088518 7433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:04.088655 7433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:04.088757 7433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:04.088833 7433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:04.099721 7433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:35281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:43009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:04.101190 7433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:04.102974 7433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:04.116248 7442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:04.116506 7439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:04.116750 7438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:04.118816 7433 server_base.cc:1034] running on GCE node
I20260924 16:05:04.120174 7433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:04.122255 7433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:04.123531 7433 hybrid_clock.cc:648] HybridClock initialized: now 1790265904123468 us; error 55 us; skew 500 ppm
I20260924 16:05:04.124017 7433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:04 dist-test-slave-xn4h krb5kdc[7420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265904, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:04.127691 7433 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:05:04.129248 7433 webserver.cc:460] Webserver started at http://127.16.233.190:37739/ using document root <none> and password file <none>
I20260924 16:05:04.129992 7433 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:04.130142 7433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:04.130524 7433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:04.133653 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "b9ab198be64d4bd0a623785cecbdcfdd"
format_stamp: "Formatted at 2026-09-24 16:05:04 on dist-test-slave-xn4h"
server_key: "d58fbb8b927c7549b53b45c552b8e00b"
server_key_iv: "8a1856f2fd88afe2176f376cabde3bbb"
server_key_version: "encryptionkey@0"
I20260924 16:05:04.134542 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "b9ab198be64d4bd0a623785cecbdcfdd"
format_stamp: "Formatted at 2026-09-24 16:05:04 on dist-test-slave-xn4h"
server_key: "d58fbb8b927c7549b53b45c552b8e00b"
server_key_iv: "8a1856f2fd88afe2176f376cabde3bbb"
server_key_version: "encryptionkey@0"
I20260924 16:05:04.140122 7433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:05:04.143955 7449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:04.145324 7433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 16:05:04.145519 7433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "b9ab198be64d4bd0a623785cecbdcfdd"
format_stamp: "Formatted at 2026-09-24 16:05:04 on dist-test-slave-xn4h"
server_key: "d58fbb8b927c7549b53b45c552b8e00b"
server_key_iv: "8a1856f2fd88afe2176f376cabde3bbb"
server_key_version: "encryptionkey@0"
I20260924 16:05:04.145737 7433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:04.170526 7433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:04.175218 7433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:04.175536 7433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:04.194473 7433 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:43009
I20260924 16:05:04.194514 7500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:43009 every 8 connection(s)
I20260924 16:05:04.195911 7433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:04.196856 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 7433
I20260924 16:05:04.197074 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:05:04.197602 17318 external_mini_cluster.cc:1428] Setting key ffa591a1b8565f639f116fef7892ca21
I20260924 16:05:04.200798 7501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:04.212778 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:04.199414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38559 (local address 127.16.233.190:43009)
0924 16:05:04.200194 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:04.200208 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:04.200264 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:05:04.202677 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:04.202705 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:04.203200 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:04.203625 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:04.204145 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:04.206152 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:04.207525 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:04.208507 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:04.208697 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:04.209812 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:04.209824 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:04.209843 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:04.210671 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:04.210777 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:04.210810 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:04.210947 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:04.211091 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 16:05:04.211389 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":343,"threads_started":1}
I20260924 16:05:04.214605 7501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd: Bootstrap starting.
I20260924 16:05:04.216468 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:05:04.218091 7501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:04.219359 7501 log.cc:824] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:04.222612 7501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd: No bootstrap required, opened a new log
I20260924 16:05:04.228574 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:04.221822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36893 (local address 127.16.233.190:43009)
0924 16:05:04.222008 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:04.222014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:04.222569 (+ 555us) server_negotiation.cc:407] Connection header received
0924 16:05:04.222728 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:04.222733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:04.222808 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:04.222918 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:04.223396 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:04.224476 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:04.225817 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:04.226577 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:04.226714 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:04.227758 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:04.227764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:04.227769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:04.227796 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:04.227838 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:04.227857 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:04.227958 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:04.228134 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 16:05:04.228233 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:05:04.231585 7501 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ab198be64d4bd0a623785cecbdcfdd" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43009 } }
I20260924 16:05:04.231950 7501 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:04.232019 7501 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ab198be64d4bd0a623785cecbdcfdd, State: Initialized, Role: FOLLOWER
I20260924 16:05:04.232618 7501 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ab198be64d4bd0a623785cecbdcfdd" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43009 } }
I20260924 16:05:04.232848 7501 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:04.233003 7501 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:04.233173 7501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:04.235463 7501 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ab198be64d4bd0a623785cecbdcfdd" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43009 } }
I20260924 16:05:04.236004 7501 leader_election.cc:302] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ab198be64d4bd0a623785cecbdcfdd; no voters:
I20260924 16:05:04.236614 7501 leader_election.cc:288] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:04.236785 7512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:04.237605 7512 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [term 1 LEADER]: Becoming Leader. State: Replica: b9ab198be64d4bd0a623785cecbdcfdd, State: Running, Role: LEADER
I20260924 16:05:04.238179 7512 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ab198be64d4bd0a623785cecbdcfdd" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43009 } }
I20260924 16:05:04.238978 7501 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:04.241240 7513 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ab198be64d4bd0a623785cecbdcfdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ab198be64d4bd0a623785cecbdcfdd" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43009 } } }
I20260924 16:05:04.241119 7514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ab198be64d4bd0a623785cecbdcfdd. Latest consensus state: current_term: 1 leader_uuid: "b9ab198be64d4bd0a623785cecbdcfdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ab198be64d4bd0a623785cecbdcfdd" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 43009 } } }
I20260924 16:05:04.241593 7513 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:04.241657 7514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:04.245347 7520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:04.251032 7520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:04.260373 7520 catalog_manager.cc:1347] Generated new cluster ID: 5f231903c8a6439e97264cb48ec58c01
I20260924 16:05:04.260496 7520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:04.279513 7520 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:04.280822 7520 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:04.296659 7520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ab198be64d4bd0a623785cecbdcfdd: Generated new TSK 0
I20260924 16:05:04.297798 7520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:05:04.317900 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7433
W20260924 16:05:04.319108 7496 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38559
W20260924 16:05:04.319805 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7433: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:04.319957 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 7433
2026-09-24T16:05:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:04 dist-test-slave-xn4h krb5kdc[7533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:04 dist-test-slave-xn4h krb5kdc[7533](info): set up 2 sockets
Sep 24 16:05:04 dist-test-slave-xn4h krb5kdc[7533](info): commencing operation
krb5kdc: starting...
W20260924 16:05:05.432111 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:05:05 dist-test-slave-xn4h krb5kdc[7533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:05:05.695196 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:35345
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:35345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:06.035331 7547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:06.035615 7547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:06.035701 7547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:06.042984 7547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:06.043116 7547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:06.043226 7547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:06.043329 7547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:06.043409 7547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:06.053262 7547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:35345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:35345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:06.054591 7547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:06.055989 7547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:06.067632 7555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:06.067670 7552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:06.067693 7553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:06.068476 7547 server_base.cc:1034] running on GCE node
I20260924 16:05:06.069234 7547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:06.070664 7547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:06.071861 7547 hybrid_clock.cc:648] HybridClock initialized: now 1790265906071801 us; error 52 us; skew 500 ppm
I20260924 16:05:06.072197 7547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:06 dist-test-slave-xn4h krb5kdc[7533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265906, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:06.075328 7547 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:05:06.076820 7547 webserver.cc:460] Webserver started at http://127.16.233.190:44525/ using document root <none> and password file <none>
I20260924 16:05:06.077512 7547 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:06.077662 7547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:06.077977 7547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:06.080999 7547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "d9ff6a1d7df6495a9afb168f8944a774"
format_stamp: "Formatted at 2026-09-24 16:05:06 on dist-test-slave-xn4h"
server_key: "5ee630a47f6619cdcd42e3f15da797ab"
server_key_iv: "39617146889d2596853514e1581b7396"
server_key_version: "encryptionkey@0"
I20260924 16:05:06.081755 7547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "d9ff6a1d7df6495a9afb168f8944a774"
format_stamp: "Formatted at 2026-09-24 16:05:06 on dist-test-slave-xn4h"
server_key: "5ee630a47f6619cdcd42e3f15da797ab"
server_key_iv: "39617146889d2596853514e1581b7396"
server_key_version: "encryptionkey@0"
I20260924 16:05:06.086850 7547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:05:06.090626 7562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:06.091943 7547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:05:06.092135 7547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "d9ff6a1d7df6495a9afb168f8944a774"
format_stamp: "Formatted at 2026-09-24 16:05:06 on dist-test-slave-xn4h"
server_key: "5ee630a47f6619cdcd42e3f15da797ab"
server_key_iv: "39617146889d2596853514e1581b7396"
server_key_version: "encryptionkey@0"
I20260924 16:05:06.092346 7547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:06.098433 7547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:06.102680 7547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:06.103010 7547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:06.123478 7547 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:35345
I20260924 16:05:06.123486 7615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:35345 every 8 connection(s)
I20260924 16:05:06.125314 7547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:06.129225 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:06.129954 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 7547
I20260924 16:05:06.130141 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:05:06.130765 17318 external_mini_cluster.cc:1428] Setting key 74cc1a8e554c33e7e768c9db778dbd81
I20260924 16:05:06.139633 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:06.132677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51001 (local address 127.16.233.190:35345)
0924 16:05:06.133599 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:06.133618 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:06.133667 (+ 49us) server_negotiation.cc:407] Connection header received
0924 16:05:06.136052 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:06.136081 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:06.136449 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:06.136813 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:06.137929 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:06.137935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:06.137949 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:06.137963 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:06.138026 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:06.138046 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:06.138212 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:06.138446 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 16:05:06.138669 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":430,"threads_started":1}
I20260924 16:05:06.143155 7616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774: Bootstrap starting.
I20260924 16:05:06.143488 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:05:06.146257 7616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:06.147300 7616 log.cc:824] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:06.150625 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:06.147718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41761 (local address 127.16.233.190:35345)
0924 16:05:06.147946 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:06.147954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:06.148468 (+ 514us) server_negotiation.cc:407] Connection header received
0924 16:05:06.148669 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:06.148672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:06.148725 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:06.148971 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:06.149927 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:06.149932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:06.149937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:06.149954 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:06.150002 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:06.150017 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:06.150139 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:06.150264 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 16:05:06.150357 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:05:06.151611 7616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774: No bootstrap required, opened a new log
I20260924 16:05:06.159392 7616 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ff6a1d7df6495a9afb168f8944a774" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35345 } }
I20260924 16:05:06.159754 7616 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:06.159847 7616 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ff6a1d7df6495a9afb168f8944a774, State: Initialized, Role: FOLLOWER
I20260924 16:05:06.160326 7616 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ff6a1d7df6495a9afb168f8944a774" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35345 } }
I20260924 16:05:06.160565 7616 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:06.160709 7616 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:06.160928 7616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:06.163343 7616 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ff6a1d7df6495a9afb168f8944a774" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35345 } }
I20260924 16:05:06.163867 7616 leader_election.cc:302] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9ff6a1d7df6495a9afb168f8944a774; no voters:
I20260924 16:05:06.164462 7616 leader_election.cc:288] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:06.164633 7626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:06.165303 7626 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [term 1 LEADER]: Becoming Leader. State: Replica: d9ff6a1d7df6495a9afb168f8944a774, State: Running, Role: LEADER
I20260924 16:05:06.165931 7626 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ff6a1d7df6495a9afb168f8944a774" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35345 } }
I20260924 16:05:06.166618 7616 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:06.168535 7627 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9ff6a1d7df6495a9afb168f8944a774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ff6a1d7df6495a9afb168f8944a774" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35345 } } }
I20260924 16:05:06.168910 7627 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:06.169782 7631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:06.169657 7628 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9ff6a1d7df6495a9afb168f8944a774. Latest consensus state: current_term: 1 leader_uuid: "d9ff6a1d7df6495a9afb168f8944a774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ff6a1d7df6495a9afb168f8944a774" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 35345 } } }
I20260924 16:05:06.170905 7628 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:06.175680 7631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:06.183729 7631 catalog_manager.cc:1347] Generated new cluster ID: 0bb5f614edbb45bd86fbbc45f553a999
I20260924 16:05:06.184248 7631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:06.207177 7631 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:06.207633 7631 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:06.214687 7631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9ff6a1d7df6495a9afb168f8944a774: Generated new TSK 0
I20260924 16:05:06.215444 7631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:05:06.227612 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7547
W20260924 16:05:06.229265 7611 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51001
W20260924 16:05:06.230006 17318 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7547: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:06.230155 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 7547
2026-09-24T16:05:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1912 ms)
[----------] 18 tests from AuthTokenIssuingTest (35601 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:06 dist-test-slave-xn4h krb5kdc[7647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:06 dist-test-slave-xn4h krb5kdc[7647](info): set up 2 sockets
Sep 24 16:05:06 dist-test-slave-xn4h krb5kdc[7647](info): commencing operation
krb5kdc: starting...
W20260924 16:05:07.348248 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:05:07 dist-test-slave-xn4h krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:05:07.603938 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:33917
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:33917,127.16.233.189:37219,127.16.233.188:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:07.925659 7660 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:05:07.926003 7660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:07.926124 7660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:07.926273 7660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:07.933301 7660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:07.933418 7660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:07.933521 7660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:07.933617 7660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:07.933702 7660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:07.943104 7660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:33917,127.16.233.189:37219,127.16.233.188:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:33917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:07.944363 7660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:07.945660 7660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:07.957073 7665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:07.957103 7666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:07.957073 7668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:07.958252 7660 server_base.cc:1034] running on GCE node
I20260924 16:05:07.958995 7660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:07.960270 7660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:07.961508 7660 hybrid_clock.cc:648] HybridClock initialized: now 1790265907961455 us; error 30 us; skew 500 ppm
I20260924 16:05:07.961829 7660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:07 dist-test-slave-xn4h krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265907, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:07.973097 7660 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:05:07.974726 7660 webserver.cc:460] Webserver started at http://127.16.233.190:37877/ using document root <none> and password file <none>
I20260924 16:05:07.975440 7660 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:07.975569 7660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:07.975894 7660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:07.978813 7660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "37136b11b08f4ca5a67290aec616e901"
format_stamp: "Formatted at 2026-09-24 16:05:07 on dist-test-slave-xn4h"
server_key: "f8adc76dcf2bfc17c916db81b17212e2"
server_key_iv: "1bd9928a16c93356ea3d45f5567580aa"
server_key_version: "encryptionkey@0"
I20260924 16:05:07.979579 7660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "37136b11b08f4ca5a67290aec616e901"
format_stamp: "Formatted at 2026-09-24 16:05:07 on dist-test-slave-xn4h"
server_key: "f8adc76dcf2bfc17c916db81b17212e2"
server_key_iv: "1bd9928a16c93356ea3d45f5567580aa"
server_key_version: "encryptionkey@0"
I20260924 16:05:07.984522 7660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:05:07.987787 7675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:07.989101 7660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:05:07.989286 7660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "37136b11b08f4ca5a67290aec616e901"
format_stamp: "Formatted at 2026-09-24 16:05:07 on dist-test-slave-xn4h"
server_key: "f8adc76dcf2bfc17c916db81b17212e2"
server_key_iv: "1bd9928a16c93356ea3d45f5567580aa"
server_key_version: "encryptionkey@0"
I20260924 16:05:07.989508 7660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:08.014518 7660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:08.018391 7660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:08.018667 7660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:08.036890 7660 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:33917
I20260924 16:05:08.036934 7728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:33917 every 8 connection(s)
I20260924 16:05:08.038221 7660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:08.041765 7729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:08.048414 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 7660
I20260924 16:05:08.048620 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:05:08.048157 7729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:05:08.049394 17318 external_mini_cluster.cc:1428] Setting key d287ed47e501d63de33cf1ab9b5838c8
Sep 24 16:05:08 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265907, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:05:08.071695 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.051605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:33917 (local address 127.0.0.1:56394)
0924 16:05:08.052365 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.052418 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:08.055182 (+ 2764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:08.056024 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:08.056033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:08.056856 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:08.057349 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:08.057374 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.058904 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.058912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:08.060076 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:08.060089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.060992 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.060999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:08.061200 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:08.061857 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:08.061883 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:08.064030 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:08.067347 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:08.067357 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:08.067367 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:08.067860 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:08.068316 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:08.068319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:08.068321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:08.068459 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:08.069242 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:08.069255 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:08.069478 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 16:05:08.069721 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 16:05:08.070351 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":138,"thread_start_us":295,"threads_started":1}
I20260924 16:05:08.071732 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.052569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.16.233.190:33917)
0924 16:05:08.053210 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:08.053216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:08.053252 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:05:08.055563 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:08.055569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:08.055771 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:08.055925 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:08.057554 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.058754 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.060223 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.060764 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.060959 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:08.064241 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:08.064248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:08.064258 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:08.064296 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:08.067200 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:08.067990 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:08.067998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:08.068002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:08.068180 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:08.068565 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:08.068572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:08.068575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:08.069079 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:08.069319 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:08.069832 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 16:05:08.070176 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":317,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.189@KRBTEST.COM; defaulting to no policy
I20260924 16:05:08.074972 7729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:05:08.076064 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.075710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:49436)
0924 16:05:08.075861 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.075959 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:05:08.076419 7679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.189@KRBTEST.COM" created.
I20260924 16:05:08.078042 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.077700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:49448)
0924 16:05:08.077856 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.077924 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:05:08.078361 7729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:37219: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab.
Entry for principal kudu/127.16.233.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab.
I20260924 16:05:08.105618 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.189:37219
--webserver_interface=127.16.233.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab
--principal=kudu/127.16.233.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:33917,127.16.233.189:37219,127.16.233.188:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:08.127676 7729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } attempt: 1
I20260924 16:05:08.129050 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.128397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:49452)
0924 16:05:08.128664 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.128817 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 16:05:08.130250 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.129783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:49460)
0924 16:05:08.129973 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.130070 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:05:08.130488 7729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:37219: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
I20260924 16:05:08.198769 7729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } attempt: 2
I20260924 16:05:08.200189 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.199518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:49476)
0924 16:05:08.199855 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.200007 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 16:05:08.201352 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.200939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:49484)
0924 16:05:08.201097 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.201198 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:05:08.201581 7729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:37219: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
I20260924 16:05:08.292865 7729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } attempt: 3
I20260924 16:05:08.294302 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.293667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:49500)
0924 16:05:08.293935 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.294096 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 16:05:08.295529 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.295060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:49510)
0924 16:05:08.295253 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.295347 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:05:08.295821 7729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:37219: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
W20260924 16:05:08.431671 7736 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:05:08.431972 7736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:08.432040 7736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:08.432117 7736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:08.438983 7736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:08.439112 7736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:08.439213 7736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260924 16:05:08.439100 7729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } attempt: 4
W20260924 16:05:08.439278 7736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:08.439352 7736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:08.440371 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.439733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:60118)
0924 16:05:08.440113 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.440214 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260924 16:05:08.441330 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.440979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:60124)
0924 16:05:08.441118 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.441193 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:05:08.441533 7729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:37219: Network error: Client connection negotiation failed: client connection to 127.16.233.189:37219: connect: Connection refused (error 111)
I20260924 16:05:08.449406 7736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:33917,127.16.233.189:37219,127.16.233.188:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab
--principal=kudu/127.16.233.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.189:37219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.233.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:08.450572 7736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:08.451819 7736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:08.463049 7741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:08.463115 7744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:08.463136 7742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:08.463840 7736 server_base.cc:1034] running on GCE node
I20260924 16:05:08.464469 7736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:08.465603 7736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:08.466809 7736 hybrid_clock.cc:648] HybridClock initialized: now 1790265908466752 us; error 26 us; skew 500 ppm
I20260924 16:05:08.467129 7736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:08 dist-test-slave-xn4h krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:08.470005 7736 init.cc:377] Logged in from keytab as kudu/127.16.233.189@KRBTEST.COM (short username kudu)
I20260924 16:05:08.471464 7736 webserver.cc:460] Webserver started at http://127.16.233.189:33027/ using document root <none> and password file <none>
I20260924 16:05:08.472107 7736 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:08.472239 7736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:08.472553 7736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:08.475335 7736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/data/instance:
uuid: "78f810b7f3d043e09065790f0f754906"
format_stamp: "Formatted at 2026-09-24 16:05:08 on dist-test-slave-xn4h"
server_key: "340b04ce765e96fe9b82d75fca839994"
server_key_iv: "d50907e6a88eca986b01fc9769383c04"
server_key_version: "encryptionkey@0"
I20260924 16:05:08.476104 7736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/wal/instance:
uuid: "78f810b7f3d043e09065790f0f754906"
format_stamp: "Formatted at 2026-09-24 16:05:08 on dist-test-slave-xn4h"
server_key: "340b04ce765e96fe9b82d75fca839994"
server_key_iv: "d50907e6a88eca986b01fc9769383c04"
server_key_version: "encryptionkey@0"
I20260924 16:05:08.481006 7736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:05:08.484263 7751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:08.485471 7736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:05:08.485651 7736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/wal
uuid: "78f810b7f3d043e09065790f0f754906"
format_stamp: "Formatted at 2026-09-24 16:05:08 on dist-test-slave-xn4h"
server_key: "340b04ce765e96fe9b82d75fca839994"
server_key_iv: "d50907e6a88eca986b01fc9769383c04"
server_key_version: "encryptionkey@0"
I20260924 16:05:08.485862 7736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:08.503353 7736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:08.507267 7736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:08.507589 7736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:08.525956 7736 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.189:37219
I20260924 16:05:08.525975 7804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.189:37219 every 8 connection(s)
I20260924 16:05:08.527280 7736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
I20260924 16:05:08.530859 7805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:08.531563 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 7736
I20260924 16:05:08.531744 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-1/wal/instance
I20260924 16:05:08.532183 17318 external_mini_cluster.cc:1428] Setting key 1e212ee45c74bcd4b1a8fd75e0a9b3be
I20260924 16:05:08.537221 7805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:08 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.188@KRBTEST.COM; defaulting to no policy
I20260924 16:05:08.557986 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.539882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.16.233.190:33917)
0924 16:05:08.540120 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:08.540129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:08.541201 (+ 1072us) server_negotiation.cc:407] Connection header received
0924 16:05:08.543155 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:08.543164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:08.543258 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:08.543387 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:08.544805 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.545960 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.547478 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.548228 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.548368 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:08.552274 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:08.552279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:08.552282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:08.552316 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:08.555022 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:08.555947 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:08.555950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:08.555953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:08.556068 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:08.556440 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:08.556443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:08.556444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:08.556755 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:08.556865 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:08.557569 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 16:05:08.557751 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.188@KRBTEST.COM" created.
I20260924 16:05:08.559088 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.540261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:33917 (local address 127.0.0.1:55452)
0924 16:05:08.541025 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.541075 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:08.542886 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:08.543584 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:08.543591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:08.544069 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:08.544626 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:08.544643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.546108 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.546111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:08.547335 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:08.547346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.548441 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.548446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:08.548654 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:08.549331 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:08.549353 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:08.552027 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:08.555170 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:08.555181 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:08.555200 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:08.555814 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:08.556168 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:08.556171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:08.556174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:08.556337 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:08.556880 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:08.556885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:08.557191 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 16:05:08.557407 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 16:05:08.557809 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":337,"threads_started":1}
I20260924 16:05:08.560913 7805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:08 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:05:08.575903 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.561715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:60126)
0924 16:05:08.561955 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.561976 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:08.562079 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:08.562952 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:08.562959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:08.563426 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:08.563629 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:08.563642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.564924 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.564932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:08.566465 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:08.566483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.567333 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.567339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:08.567519 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:08.568082 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:08.568098 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:08.570013 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:08.573411 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:08.573416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:08.573420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:08.573772 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:08.574158 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:08.574162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:08.574166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:08.574296 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:08.574923 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:08.574928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:08.575062 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 16:05:08.575217 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 16:05:08.575486 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 16:05:08.575978 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.561873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.16.233.189:37219)
0924 16:05:08.562555 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:08.562560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:08.562585 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:05:08.562624 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:08.562630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:08.562782 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:08.562898 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:08.563795 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.564782 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.566633 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.567165 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.567303 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:08.570221 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:08.570231 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:08.570243 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:08.570274 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:08.573252 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:08.573904 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:08.573918 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:08.573921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:08.574030 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:08.574425 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:08.574431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:08.574434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:08.574771 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:08.575001 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:08.575356 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:05:08.575575 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":376,"threads_started":1}
I20260924 16:05:08.578536 7805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:05:08.579602 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.579184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42492)
0924 16:05:08.579353 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.579462 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:05:08.579993 7754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
I20260924 16:05:08.581590 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.581244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42496)
0924 16:05:08.581408 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.581491 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 16:05:08.581881 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:35959: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab.
Entry for principal kudu/127.16.233.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab.
I20260924 16:05:08.590042 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.188:35959
--webserver_interface=127.16.233.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab
--principal=kudu/127.16.233.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:33917,127.16.233.189:37219,127.16.233.188:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:08.631206 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } attempt: 1
I20260924 16:05:08.632445 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.631990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42512)
0924 16:05:08.632181 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.632285 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 16:05:08.633421 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.633073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42526)
0924 16:05:08.633227 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.633313 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:05:08.633651 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:35959: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
I20260924 16:05:08.701859 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } attempt: 2
I20260924 16:05:08.703042 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.702582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42528)
0924 16:05:08.702786 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.702886 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 16:05:08.704035 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.703691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42536)
0924 16:05:08.703845 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.703922 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:05:08.704227 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:35959: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
I20260924 16:05:08.740773 7729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } attempt: 5
Sep 24 16:05:08 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265907, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:05:08.753887 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.741650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60134 (local address 127.16.233.189:37219)
0924 16:05:08.741851 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:08.741860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:08.741883 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:05:08.742104 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:08.742110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:08.742199 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:08.742310 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:08.743085 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.743786 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.744995 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.745465 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.745604 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:08.748707 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:08.748711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:08.748714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:08.748745 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:08.751346 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:08.751990 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:08.751997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:08.752001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:08.752152 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:08.752547 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:08.752553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:08.752556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:08.752825 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:08.752981 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:08.753357 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 16:05:08.753575 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:05:08.753908 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.741544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:60134)
0924 16:05:08.741775 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.741797 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:08.741963 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:08.742370 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:08.742375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:08.742772 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:08.742930 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:08.742939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.743923 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.743929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:08.744840 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:08.744852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:08.745616 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:08.745621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:08.745835 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:08.746499 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:08.746517 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:08.748490 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:08.751502 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:08.751507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:08.751512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:08.751862 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:08.752267 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:08.752272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:08.752276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:08.752435 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:08.752947 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:08.752951 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:08.753095 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 16:05:08.753239 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 16:05:08.753550 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 16:05:08.755172 7729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:05:08.756112 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.755761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42546)
0924 16:05:08.755919 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.755996 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 16:05:08.756983 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.756665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42548)
0924 16:05:08.756820 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.756882 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:05:08.757220 7729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:35959: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
I20260924 16:05:08.795462 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } attempt: 3
I20260924 16:05:08.796720 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.796112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42554)
0924 16:05:08.796374 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.796508 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 16:05:08.797890 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.797412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42556)
0924 16:05:08.797646 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.797742 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260924 16:05:08.798123 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:35959: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
I20260924 16:05:08.808434 7729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } attempt: 1
I20260924 16:05:08.809648 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.809055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42568)
0924 16:05:08.809292 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.809425 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 16:05:08.810794 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.810346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42578)
0924 16:05:08.810535 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.810636 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:05:08.811002 7729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:35959: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
I20260924 16:05:08.879235 7729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } attempt: 2
I20260924 16:05:08.880615 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.879961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42594)
0924 16:05:08.880250 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.880374 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 16:05:08.881728 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.881320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42608)
0924 16:05:08.881498 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.881583 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 16:05:08.881975 7729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:35959: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
W20260924 16:05:08.917235 7812 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:05:08.917598 7812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:08.917752 7812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:08.917891 7812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:08.925314 7812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:08.925400 7812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:08.925498 7812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:08.925591 7812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:08.925706 7812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:08.935593 7812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:33917,127.16.233.189:37219,127.16.233.188:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab
--principal=kudu/127.16.233.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.188:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.233.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:08.936893 7812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:08.938203 7812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:05:08.941349 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } attempt: 4
I20260924 16:05:08.942605 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.942022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42614)
0924 16:05:08.942267 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.942404 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:05:08.943687 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.943293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42628)
0924 16:05:08.943474 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.943559 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 16:05:08.944011 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:35959: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
W20260924 16:05:08.950421 7818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:08.950503 7817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:08.950470 7820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:08.951504 7812 server_base.cc:1034] running on GCE node
I20260924 16:05:08.952261 7812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:08.953681 7812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:08.954888 7812 hybrid_clock.cc:648] HybridClock initialized: now 1790265908954832 us; error 28 us; skew 500 ppm
I20260924 16:05:08.955189 7812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:08 dist-test-slave-xn4h krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:08.958171 7812 init.cc:377] Logged in from keytab as kudu/127.16.233.188@KRBTEST.COM (short username kudu)
I20260924 16:05:08.959566 7812 webserver.cc:460] Webserver started at http://127.16.233.188:39419/ using document root <none> and password file <none>
I20260924 16:05:08.960286 7812 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:08.960400 7812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:08.960750 7812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:08.963764 7812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/data/instance:
uuid: "7cea73cfc00641468632b4cbdb54e5f7"
format_stamp: "Formatted at 2026-09-24 16:05:08 on dist-test-slave-xn4h"
server_key: "b3b672e95b55b4efb118bda23b016aa4"
server_key_iv: "36188b2d4c5758cbfff2e2d202be5949"
server_key_version: "encryptionkey@0"
I20260924 16:05:08.964546 7812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/wal/instance:
uuid: "7cea73cfc00641468632b4cbdb54e5f7"
format_stamp: "Formatted at 2026-09-24 16:05:08 on dist-test-slave-xn4h"
server_key: "b3b672e95b55b4efb118bda23b016aa4"
server_key_iv: "36188b2d4c5758cbfff2e2d202be5949"
server_key_version: "encryptionkey@0"
I20260924 16:05:08.969723 7812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 16:05:08.973110 7828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:08.974551 7812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:05:08.974722 7812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/wal
uuid: "7cea73cfc00641468632b4cbdb54e5f7"
format_stamp: "Formatted at 2026-09-24 16:05:08 on dist-test-slave-xn4h"
server_key: "b3b672e95b55b4efb118bda23b016aa4"
server_key_iv: "36188b2d4c5758cbfff2e2d202be5949"
server_key_version: "encryptionkey@0"
I20260924 16:05:08.974936 7812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:08.988204 7729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } attempt: 3
I20260924 16:05:08.989368 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.988896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42642)
0924 16:05:08.989087 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.989183 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 16:05:08.990482 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:08.990063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42650)
0924 16:05:08.990272 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:08.990364 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 16:05:08.990700 7729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:35959: Network error: Client connection negotiation failed: client connection to 127.16.233.188:35959: connect: Connection refused (error 111)
I20260924 16:05:09.002151 7812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:09.006160 7812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:09.006474 7812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:09.024888 7812 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.188:35959
I20260924 16:05:09.024909 7879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.188:35959 every 8 connection(s)
I20260924 16:05:09.026206 7812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
I20260924 16:05:09.029709 7880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:09.033682 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 7812
I20260924 16:05:09.033887 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265545293861-17318-0/minicluster-data/master-2/wal/instance
I20260924 16:05:09.034277 17318 external_mini_cluster.cc:1428] Setting key 999c58c3717f9ec59b329788112b408e
I20260924 16:05:09.036528 7880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:09 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
Sep 24 16:05:09 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:05:09.057521 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.041159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.16.233.190:33917)
0924 16:05:09.041346 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.041352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.042043 (+ 691us) server_negotiation.cc:407] Connection header received
0924 16:05:09.042331 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.042337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.042422 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.042525 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.043987 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.045112 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.046749 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.047214 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.047310 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.051277 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.051285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.051294 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.051332 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.054620 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.055386 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.055390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.055392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.055517 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.055885 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.055889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.055892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.056195 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.056357 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.056969 (+ 612us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.057193 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:05:09.058420 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.036106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.16.233.188:35959)
0924 16:05:09.037013 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.037023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.037061 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:05:09.039184 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.039204 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.039595 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.039970 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.041772 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.043392 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.045069 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.045796 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.045930 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.050682 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.050689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.050700 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.050732 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.054225 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.055053 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.055059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.055061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.055212 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.055607 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.055612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.055616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.056231 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.056493 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.056825 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.057264 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":697,"thread_start_us":484,"threads_started":1}
I20260924 16:05:09.058476 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.041131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:33917 (local address 127.0.0.1:55456)
0924 16:05:09.041887 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:09.041936 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:09.042163 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:09.042611 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:09.042618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:09.043507 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:09.043815 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.043838 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.045265 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.045277 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.046581 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.046594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.047450 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.047460 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.047668 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.048278 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:09.048316 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:09.051076 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:09.054737 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.054753 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.054765 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.055249 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.055648 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.055652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.055654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.055773 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.056356 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:09.056364 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:09.056594 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 16:05:09.056865 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 16:05:09.057430 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":141,"thread_start_us":351,"threads_started":1}
I20260924 16:05:09.060761 7880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:09 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
Sep 24 16:05:09 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:05:09.074640 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.061548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:37219 (local address 127.0.0.1:60148)
0924 16:05:09.062018 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:09.062033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:09.062133 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:09.062483 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:09.062487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:09.062792 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:09.062933 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.062940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.064084 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.064090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.065088 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.065101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.065994 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.066001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.066224 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.066878 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:09.066894 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:09.068839 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:09.072187 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.072194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.072198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.072734 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.073107 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.073112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.073116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.073286 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.073774 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:09.073779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:09.073955 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 16:05:09.074110 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 16:05:09.074367 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370}
I20260924 16:05:09.074946 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.061529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.16.233.189:37219)
0924 16:05:09.061891 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.061898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.062109 (+ 211us) server_negotiation.cc:407] Connection header received
0924 16:05:09.062252 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.062256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.062354 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.062471 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.063073 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.063944 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.065229 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.065857 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.065950 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.069032 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.069037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.069041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.069071 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.072024 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.072863 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.072867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.072869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.072993 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.073394 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.073397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.073399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.073650 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.073765 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.074248 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.074505 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260924 16:05:09.076028 7880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:05:09.076761 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.061787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.16.233.190:33917)
0924 16:05:09.061967 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.061973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.061989 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:05:09.062038 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.062043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.062119 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.062213 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.063505 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.064380 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.065779 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.066428 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.066552 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.070745 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.070750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.070753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.070813 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.073887 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.074686 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.074693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.074696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.074908 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.075345 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.075350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.075352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.075692 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.075957 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.076233 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.076444 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 16:05:09 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
Sep 24 16:05:09 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:05:09.089622 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.076797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42680)
0924 16:05:09.077264 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:09.077277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:09.077394 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:09.077672 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:09.077676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:09.078076 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:09.078262 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.078278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.079723 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.079730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.080919 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.080931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.081582 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.081587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.081850 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.082451 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:09.082467 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:09.084504 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:09.087462 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.087466 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.087468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.087805 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.088139 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.088145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.088148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.088270 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.088744 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:09.088748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:09.088908 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 16:05:09.089050 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 16:05:09.089345 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":24}
I20260924 16:05:09.089967 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.076955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.16.233.188:35959)
0924 16:05:09.077093 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.077101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.077350 (+ 249us) server_negotiation.cc:407] Connection header received
0924 16:05:09.077499 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.077505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.077572 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.077658 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.078476 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.079553 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.081047 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.081444 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.081576 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.084683 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.084687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.084690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.084716 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.087342 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.087918 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.087925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.087929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.088039 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.088378 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.088384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.088387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.088620 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.088777 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.089134 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.089514 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":34}
I20260924 16:05:09.093148 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.078487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.16.233.189:37219)
0924 16:05:09.078625 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.078628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.078642 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:05:09.078717 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.078719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.078765 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.078833 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.080578 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.081247 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.082610 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.083041 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.083174 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.087230 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.087236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.087243 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.087282 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.090192 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.091074 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.091082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.091086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.091305 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.091687 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.091691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.091694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.092061 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.092319 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.092579 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.092818 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 16:05:09.096092 7880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7: Bootstrap starting.
I20260924 16:05:09.097805 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:05:09.099694 7880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:09.101125 7880 log.cc:824] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:09.100489 7704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "37136b11b08f4ca5a67290aec616e901"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55470
I20260924 16:05:09.104336 7880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7: No bootstrap required, opened a new log
I20260924 16:05:09.115015 7880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } }
I20260924 16:05:09.115420 7880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cea73cfc00641468632b4cbdb54e5f7, State: Initialized, Role: FOLLOWER
I20260924 16:05:09.115904 7880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } }
I20260924 16:05:09.116896 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.117228 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:09.118100 7880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:05:09.128134 7897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:05:09.128254 7897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:05:09.126847 7812 master_runner.cc:186] Error getting master registration for 127.16.233.190:33917: OK, instance_id { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" instance_seqno: 1790265908031709 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:05:09.134509 7812 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:05:09.167908 7729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } attempt: 4
Sep 24 16:05:09 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265907, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
I20260924 16:05:09.181782 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.168624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42684)
0924 16:05:09.168822 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:09.168843 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:09.168968 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:09.169300 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:09.169305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:09.169717 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:09.169889 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.169898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.171049 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.171055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.171999 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.172011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.172743 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.172749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.172955 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.173554 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:09.173571 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:09.175805 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:09.179097 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.179104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.179108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.179644 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.180028 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.180034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.180038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.180179 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.180774 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:09.180779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:09.180927 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 16:05:09.181069 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 16:05:09.181353 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 16:05:09.181782 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.168708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.16.233.188:35959)
0924 16:05:09.168953 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.168958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.168980 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:05:09.169084 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.169089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.169162 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.169264 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.170041 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.170895 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.172158 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.172587 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.172714 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.176025 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.176030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.176033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.176063 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.178943 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.179753 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.179759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.179775 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.179909 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.180288 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.180294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.180297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.180636 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.180796 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.181197 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.181414 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 16:05:09.187920 7729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901: Bootstrap starting.
I20260924 16:05:09.192004 7729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:09.193109 7729 log.cc:824] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:09.196332 7729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901: No bootstrap required, opened a new log
I20260924 16:05:09.203704 7729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } }
I20260924 16:05:09.204097 7729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37136b11b08f4ca5a67290aec616e901, State: Initialized, Role: FOLLOWER
I20260924 16:05:09.204502 7729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } }
I20260924 16:05:09.205468 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.205838 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:09.206571 7729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:09.206677 7704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:05:09.206902 7704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:09.211889 7704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } }
I20260924 16:05:09.214257 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37136b11b08f4ca5a67290aec616e901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7cea73cfc00641468632b4cbdb54e5f7"
I20260924 16:05:09.214795 7855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:09.218107 7855 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37136b11b08f4ca5a67290aec616e901 in term 1.
I20260924 16:05:09.218482 7704 leader_election.cc:288] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [CANDIDATE]: Term 1 election: Requested vote from peers 78f810b7f3d043e09065790f0f754906 (127.16.233.189:37219), 7cea73cfc00641468632b4cbdb54e5f7 (127.16.233.188:35959)
I20260924 16:05:09.218724 7678 leader_election.cc:302] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37136b11b08f4ca5a67290aec616e901, 7cea73cfc00641468632b4cbdb54e5f7; no voters:
I20260924 16:05:09.219554 7900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:09.220650 7900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [term 1 LEADER]: Becoming Leader. State: Replica: 37136b11b08f4ca5a67290aec616e901, State: Running, Role: LEADER
I20260924 16:05:09.221237 7900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } }
I20260924 16:05:09.226020 7911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37136b11b08f4ca5a67290aec616e901. Latest consensus state: current_term: 1 leader_uuid: "37136b11b08f4ca5a67290aec616e901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.226620 7911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [sys.catalog]: This master's current role is: LEADER
W20260924 16:05:09.238530 7914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:05:09.240595 7914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:05:09.243251 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } attempt: 5
I20260924 16:05:09.246383 7915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:09.250692 7915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 16:05:09 dist-test-slave-xn4h krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
W20260924 16:05:09.258528 7834 tablet.cc:2369] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:09.236734 7660 master_runner.cc:186] Error getting master registration for 127.16.233.189:37219: OK, instance_id { permanent_uuid: "78f810b7f3d043e09065790f0f754906" instance_seqno: 1790265908520763 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:05:09.261053 7660 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:05:09.267400 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.243824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:35959 (local address 127.0.0.1:42688)
0924 16:05:09.244202 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:09.244224 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:09.244359 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:09.245279 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:09.245285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:09.245830 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:09.246057 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.246070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.247547 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.247552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.249005 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:09.249021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.250219 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.250228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:09.250509 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.251244 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:09.251283 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:09.254080 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:09.262326 (+ 8246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.262334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.262341 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.263034 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.263961 (+ 927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:09.263969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:09.263976 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:09.264215 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:09.266095 (+ 1880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:09.266115 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:09.266361 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 16:05:09.266521 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 16:05:09.266825 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
W20260924 16:05:09.271468 7681 tablet.cc:2369] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:09.272393 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.244584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.16.233.188:35959)
0924 16:05:09.244762 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.244768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.244786 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:05:09.244851 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.244856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.244951 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.245055 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.246242 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.247371 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.249196 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.249884 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.250007 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.254337 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.254347 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.254355 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.254409 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.262031 (+ 7622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.263331 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.263339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.263345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.263704 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.264468 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.264476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.264482 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.265690 (+ 1208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.268348 (+ 2658us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.269110 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.270369 (+ 1259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:05:09.281273 7805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906: Bootstrap starting.
I20260924 16:05:09.291740 7805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:09.296401 7805 log.cc:824] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:09.317219 7855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [term 1 FOLLOWER]: Refusing update from remote peer 37136b11b08f4ca5a67290aec616e901: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:09.318396 7911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:09.330660 7805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906: No bootstrap required, opened a new log
I20260924 16:05:09.353937 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37136b11b08f4ca5a67290aec616e901. Latest consensus state: current_term: 1 leader_uuid: "37136b11b08f4ca5a67290aec616e901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.354298 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:09.366015 7919 mvcc.cc:205] Tried to move back new op lower bound from 7332929164539207680 to 7332929164176326656. Current Snapshot: MvccSnapshot[applied={T|T < 7332929164539207680}]
I20260924 16:05:09.372987 7924 mvcc.cc:205] Tried to move back new op lower bound from 7332929164539207680 to 7332929164176326656. Current Snapshot: MvccSnapshot[applied={T|T < 7332929164539207680}]
I20260924 16:05:09.374933 7805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } }
I20260924 16:05:09.375593 7805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f810b7f3d043e09065790f0f754906, State: Initialized, Role: FOLLOWER
I20260924 16:05:09.376426 7805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } }
I20260924 16:05:09.378846 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.316862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.16.233.189:37219)
0924 16:05:09.317130 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.317137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.324202 (+ 7065us) server_negotiation.cc:407] Connection header received
0924 16:05:09.325035 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.325042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.325138 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.325214 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.336564 (+ 11350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.337769 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.348442 (+ 10673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.349162 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.349418 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.363886 (+ 14468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.363893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.363897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.363943 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.367842 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.372289 (+ 4447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.372298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.372305 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.372509 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.374626 (+ 2117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.374633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.374638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.375093 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.375396 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.378023 (+ 2627us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.378358 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 16:05:09.389292 7930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.389876 7930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:09.391781 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37136b11b08f4ca5a67290aec616e901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "78f810b7f3d043e09065790f0f754906"
I20260924 16:05:09.392454 7779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:09.393132 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37136b11b08f4ca5a67290aec616e901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.393392 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:09.396121 7779 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [term 1 FOLLOWER]: Refusing update from remote peer 37136b11b08f4ca5a67290aec616e901: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:09.397424 7900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:09.401595 7780 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37136b11b08f4ca5a67290aec616e901 in term 1.
I20260924 16:05:09.402952 7915 catalog_manager.cc:1347] Generated new cluster ID: 36e56c9ea57d45fe87eaf24832cd378c
I20260924 16:05:09.403059 7915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:09.404497 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.316789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.16.233.190:33917)
0924 16:05:09.316962 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.316971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.333898 (+ 16927us) server_negotiation.cc:407] Connection header received
0924 16:05:09.335076 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.335082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.335173 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.335275 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.351572 (+ 16297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.352788 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.360317 (+ 7529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.361085 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.361198 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.384267 (+ 23069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.384284 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.384288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.384338 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.388568 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.391703 (+ 3135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.391708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.391712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.391895 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.402452 (+ 10557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.402460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.402466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.402953 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.403224 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.403818 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.404135 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 16:05:09.392300 7805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:09.418118 7931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37136b11b08f4ca5a67290aec616e901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.418601 7931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:09.429612 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.325459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.16.233.188:35959)
0924 16:05:09.325700 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.325708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.363104 (+ 37396us) server_negotiation.cc:407] Connection header received
0924 16:05:09.366506 (+ 3402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.366514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.366617 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.366724 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.384260 (+ 17536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.385360 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.393965 (+ 8605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.394731 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.394842 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.418513 (+ 23671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:09.418520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:09.418526 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:09.418582 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:09.422637 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.427105 (+ 4468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.427116 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.427120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.427271 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:09.427759 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:09.427765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:09.427768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:09.428146 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:09.428410 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.428983 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.429237 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:05:09.434166 7930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37136b11b08f4ca5a67290aec616e901. Latest consensus state: current_term: 1 leader_uuid: "37136b11b08f4ca5a67290aec616e901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.434536 7930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:09.434342 7938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37136b11b08f4ca5a67290aec616e901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.435111 7938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:09.435842 7931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37136b11b08f4ca5a67290aec616e901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37136b11b08f4ca5a67290aec616e901" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33917 } } peers { permanent_uuid: "78f810b7f3d043e09065790f0f754906" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 37219 } } peers { permanent_uuid: "7cea73cfc00641468632b4cbdb54e5f7" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 35959 } } }
I20260924 16:05:09.436095 7931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:09.442555 7915 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:09.444509 7915 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:09.468856 7915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37136b11b08f4ca5a67290aec616e901: Generated new TSK 0
I20260924 16:05:09.470072 7915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:05:09.507675 7946 catalog_manager.cc:1259] Loaded cluster ID: 36e56c9ea57d45fe87eaf24832cd378c
I20260924 16:05:09.507917 7946 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906: loading cluster ID for follower catalog manager: success
I20260924 16:05:09.512441 7946 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906: acquiring CA information for follower catalog manager: success
I20260924 16:05:09.515785 7946 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78f810b7f3d043e09065790f0f754906: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 16:05:09.633432 7732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.621618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.16.233.190:33917)
0924 16:05:09.621846 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.621857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.630920 (+ 9063us) server_negotiation.cc:407] Connection header received
0924 16:05:09.631925 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.631932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.632033 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.632103 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.633167 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55490: BlockingRecv error: recv got EOF from 127.0.0.1:55490 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 16:05:09.638386 7809 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.625601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.16.233.189:37219)
0924 16:05:09.625787 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.625795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.634915 (+ 9120us) server_negotiation.cc:407] Connection header received
0924 16:05:09.636915 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.636922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.637033 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.637111 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.638126 (+ 1015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60166: BlockingRecv error: recv got EOF from 127.0.0.1:60166 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 16:05:09.640800 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.636041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.16.233.188:35959)
0924 16:05:09.636215 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.636220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.638983 (+ 2763us) server_negotiation.cc:407] Connection header received
0924 16:05:09.639231 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.639238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.639321 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.639394 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.640553 (+ 1159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42696: BlockingRecv error: recv got EOF from 127.0.0.1:42696 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:05:09.686928 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.671270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.16.233.189:37219)
0924 16:05:09.671483 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.671489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.678590 (+ 7101us) server_negotiation.cc:407] Connection header received
0924 16:05:09.678953 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.678961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.679181 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.679343 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:09.680123 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.681343 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.683804 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.684488 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.684587 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.685349 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:09.686064 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:09.686191 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.686468 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.686570 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 16:05:09.731765 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.721692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.16.233.188:35959)
0924 16:05:09.721892 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.721899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.729430 (+ 7531us) server_negotiation.cc:407] Connection header received
0924 16:05:09.730052 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.730060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.730192 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.730277 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.731511 (+ 1234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42698: BlockingRecv error: recv got EOF from 127.0.0.1:42698 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:05:09.771590 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.762458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.16.233.189:37219)
0924 16:05:09.762653 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.762659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.764997 (+ 2338us) server_negotiation.cc:407] Connection header received
0924 16:05:09.765197 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.765207 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.765352 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.765479 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:09.766056 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.767338 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.769340 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.769988 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.770150 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.770705 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:09.770894 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:09.770999 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.771164 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.771248 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 16:05:09.788502 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.783880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.16.233.188:35959)
0924 16:05:09.784104 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.784110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.786897 (+ 2787us) server_negotiation.cc:407] Connection header received
0924 16:05:09.787097 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.787103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.787206 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.787325 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.788249 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42702: BlockingRecv error: recv got EOF from 127.0.0.1:42702 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:05:09.801378 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.794403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.16.233.189:37219)
0924 16:05:09.794596 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.794602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.795228 (+ 626us) server_negotiation.cc:407] Connection header received
0924 16:05:09.795464 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.795471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.795596 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.795718 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:09.796166 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.797368 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.799239 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.799899 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.800037 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.800508 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:09.800665 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:09.800772 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.800948 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.801040 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 16:05:09.824383 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.820008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.16.233.188:35959)
0924 16:05:09.820215 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.820221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.822870 (+ 2649us) server_negotiation.cc:407] Connection header received
0924 16:05:09.823060 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.823066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.823173 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.823296 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.824126 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42706: BlockingRecv error: recv got EOF from 127.0.0.1:42706 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:05:09.839217 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.832404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.16.233.189:37219)
0924 16:05:09.832625 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.832632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.833132 (+ 500us) server_negotiation.cc:407] Connection header received
0924 16:05:09.833315 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.833321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.833411 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.833527 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:09.834011 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.835274 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.837158 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.837865 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.837984 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.838506 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:09.838632 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:09.838720 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.838888 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.838970 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 16:05:09.852347 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.848093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.16.233.188:35959)
0924 16:05:09.848330 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.848336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.850854 (+ 2518us) server_negotiation.cc:407] Connection header received
0924 16:05:09.851030 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.851036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.851153 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.851265 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.852107 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42718: BlockingRecv error: recv got EOF from 127.0.0.1:42718 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
I20260924 16:05:09.873641 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.866730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.16.233.189:37219)
0924 16:05:09.866957 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.866963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.867499 (+ 536us) server_negotiation.cc:407] Connection header received
0924 16:05:09.867671 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.867677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.867781 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.867881 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:09.868320 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.869490 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.871459 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.872142 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.872290 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.872729 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:09.872867 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:09.872963 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.873220 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.873313 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 16:05:09.882546 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.880423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.16.233.188:35959)
0924 16:05:09.880600 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.880606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.881142 (+ 536us) server_negotiation.cc:407] Connection header received
0924 16:05:09.881311 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.881316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.881402 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.881506 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.882343 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42728: BlockingRecv error: recv got EOF from 127.0.0.1:42728 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:05:09.908169 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.902352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.16.233.189:37219)
0924 16:05:09.902559 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.902566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.902906 (+ 340us) server_negotiation.cc:407] Connection header received
0924 16:05:09.903074 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.903081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.903181 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.903319 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:09.903685 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.904908 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.906184 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.906907 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.907056 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.907445 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:09.907577 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:09.907677 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.907797 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.907885 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 16:05:09.916797 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.914818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.16.233.188:35959)
0924 16:05:09.915050 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.915056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.915336 (+ 280us) server_negotiation.cc:407] Connection header received
0924 16:05:09.915521 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.915525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.915605 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.915680 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.916566 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42738: BlockingRecv error: recv got EOF from 127.0.0.1:42738 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 16:05:09.958474 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:09.952478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.16.233.189:37219)
0924 16:05:09.952692 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.952698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.953019 (+ 321us) server_negotiation.cc:407] Connection header received
0924 16:05:09.953174 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.953180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.953321 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.953436 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:09.953805 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.955025 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.956341 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:09.957142 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:09.957283 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:09.957666 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:09.957800 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:09.957915 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:09.958027 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 16:05:09.958108 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260924 16:05:09.967185 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:09.965083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.16.233.188:35959)
0924 16:05:09.965296 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:09.965301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:09.965689 (+ 388us) server_negotiation.cc:407] Connection header received
0924 16:05:09.965863 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:09.965867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:09.965943 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:09.966022 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:09.966935 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42746: BlockingRecv error: recv got EOF from 127.0.0.1:42746 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:05:10.041283 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:10.034731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.16.233.189:37219)
0924 16:05:10.034941 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:10.034947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:10.035357 (+ 410us) server_negotiation.cc:407] Connection header received
0924 16:05:10.035496 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:10.035501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:10.035581 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:10.035684 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:10.036164 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.037267 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.039279 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.039924 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.040046 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:10.040515 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:10.040641 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:10.040728 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:10.040907 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 16:05:10.040998 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 16:05:10.050086 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:10.048192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.16.233.188:35959)
0924 16:05:10.048355 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:10.048359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:10.048848 (+ 489us) server_negotiation.cc:407] Connection header received
0924 16:05:10.048983 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:10.048991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:10.049103 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:10.049209 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:10.049878 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42748: BlockingRecv error: recv got EOF from 127.0.0.1:42748 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:05:10.130127 7897 catalog_manager.cc:1259] Loaded cluster ID: 36e56c9ea57d45fe87eaf24832cd378c
I20260924 16:05:10.130219 7897 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7: loading cluster ID for follower catalog manager: success
I20260924 16:05:10.132673 7897 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7: acquiring CA information for follower catalog manager: success
I20260924 16:05:10.134436 7897 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7cea73cfc00641468632b4cbdb54e5f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:05:10.187112 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:10.181978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.16.233.189:37219)
0924 16:05:10.182235 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:10.182241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:10.182499 (+ 258us) server_negotiation.cc:407] Connection header received
0924 16:05:10.182760 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:10.182764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:10.182832 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:10.182937 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:10.183385 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.184082 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.185431 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.185827 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.185966 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:10.186372 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:10.186497 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:10.186598 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:10.186747 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 16:05:10.186831 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:05:10.197996 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:10.192821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.16.233.188:35959)
0924 16:05:10.193051 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:10.193055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:10.193214 (+ 159us) server_negotiation.cc:407] Connection header received
0924 16:05:10.193364 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:10.193368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:10.193523 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:10.193628 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:10.193995 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.194756 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.195985 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.196354 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.196450 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:10.196899 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:10.197403 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:10.197520 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:10.197628 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 16:05:10.197692 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:05:10.211392 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:10.204512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.16.233.189:37219)
0924 16:05:10.204775 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:10.204784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:10.206367 (+ 1583us) server_negotiation.cc:407] Connection header received
0924 16:05:10.206585 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:10.206591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:10.206671 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:10.206768 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:10.207305 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.208045 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.209576 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.210007 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.210153 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:10.210690 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:10.210796 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:10.210901 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:10.211023 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 16:05:10.211104 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:05:10.216352 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:10.205461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.16.233.188:35959)
0924 16:05:10.205628 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:10.205634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:10.211352 (+ 5718us) server_negotiation.cc:407] Connection header received
0924 16:05:10.211724 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:10.211729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:10.211811 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:10.211868 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:10.212321 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.213025 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.214329 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.214960 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.215103 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:10.215581 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:10.215717 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:10.215819 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:10.216010 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 16:05:10.216096 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 16:05:10.220857 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:10.204317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.16.233.190:33917)
0924 16:05:10.214703 (+ 10386us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:10.214710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:10.214730 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:05:10.214850 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:10.214855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:10.215367 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:10.215602 (+ 235us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:10.216305 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.217372 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.218815 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:10.219365 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:10.219471 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:10.219906 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:10.220250 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:10.220332 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:10.220489 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 16:05:10.220585 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10252,"thread_start_us":10140,"threads_started":1}
I20260924 16:05:10.226094 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7660
I20260924 16:05:10.227192 7724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:10.336102 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 7660
I20260924 16:05:10.357390 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7736
I20260924 16:05:10.358513 7800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:10.464147 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 7736
I20260924 16:05:10.484503 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 7812
I20260924 16:05:10.485796 7875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:10.590996 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 7812
2026-09-24T16:05:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4365 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:10 dist-test-slave-xn4h krb5kdc[8061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:10 dist-test-slave-xn4h krb5kdc[8061](info): set up 2 sockets
Sep 24 16:05:10 dist-test-slave-xn4h krb5kdc[8061](info): commencing operation
krb5kdc: starting...
W20260924 16:05:11.704679 17318 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/test-user.keytab.
Sep 24 16:05:11 dist-test-slave-xn4h krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
Entry for principal kudu/127.16.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab.
I20260924 16:05:11.954981 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:33939
--webserver_interface=127.16.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:33939,127.16.233.189:38739,127.16.233.188:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:12.278601 8074 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:05:12.278923 8074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:12.279004 8074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:12.279086 8074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:12.286360 8074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:12.286451 8074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:12.286509 8074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:12.286552 8074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:12.286621 8074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:12.296262 8074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:33939,127.16.233.189:38739,127.16.233.188:42775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.190.keytab
--principal=kudu/127.16.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.190:33939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:12.297459 8074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:12.298727 8074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:12.310187 8082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:12.310391 8079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:12.310294 8080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:12.311183 8074 server_base.cc:1034] running on GCE node
I20260924 16:05:12.311843 8074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:12.313233 8074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:12.314433 8074 hybrid_clock.cc:648] HybridClock initialized: now 1790265912314359 us; error 31 us; skew 500 ppm
I20260924 16:05:12.314797 8074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:12 dist-test-slave-xn4h krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:12.317860 8074 init.cc:377] Logged in from keytab as kudu/127.16.233.190@KRBTEST.COM (short username kudu)
I20260924 16:05:12.319242 8074 webserver.cc:460] Webserver started at http://127.16.233.190:46603/ using document root <none> and password file <none>
I20260924 16:05:12.319957 8074 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:12.320091 8074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:12.320395 8074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:12.323269 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/data/instance:
uuid: "389b344632224d249eabd0208f7dae0c"
format_stamp: "Formatted at 2026-09-24 16:05:12 on dist-test-slave-xn4h"
server_key: "6c3f8f23ae3bc8a430e011fa1a6d6ab2"
server_key_iv: "0c260193a7300a35dd0fc0d69f9746d8"
server_key_version: "encryptionkey@0"
I20260924 16:05:12.323992 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/wal/instance:
uuid: "389b344632224d249eabd0208f7dae0c"
format_stamp: "Formatted at 2026-09-24 16:05:12 on dist-test-slave-xn4h"
server_key: "6c3f8f23ae3bc8a430e011fa1a6d6ab2"
server_key_iv: "0c260193a7300a35dd0fc0d69f9746d8"
server_key_version: "encryptionkey@0"
I20260924 16:05:12.328847 8074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:05:12.332162 8089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:12.333344 8074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:05:12.333549 8074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/wal
uuid: "389b344632224d249eabd0208f7dae0c"
format_stamp: "Formatted at 2026-09-24 16:05:12 on dist-test-slave-xn4h"
server_key: "6c3f8f23ae3bc8a430e011fa1a6d6ab2"
server_key_iv: "0c260193a7300a35dd0fc0d69f9746d8"
server_key_version: "encryptionkey@0"
I20260924 16:05:12.333774 8074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:12.378652 8074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:12.382608 8074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:12.382889 8074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:12.401098 8074 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.190:33939
I20260924 16:05:12.401116 8142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.190:33939 every 8 connection(s)
I20260924 16:05:12.402405 8074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:12.406049 8143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:12.407232 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 8074
I20260924 16:05:12.407415 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-0/wal/instance
I20260924 16:05:12.407840 17318 external_mini_cluster.cc:1428] Setting key 4615a5098411e28e1aca3bd030474098
I20260924 16:05:12.412681 8143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:12 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.189@KRBTEST.COM" created.
I20260924 16:05:12.438442 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.415665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:33939 (local address 127.0.0.1:60824)
0924 16:05:12.416483 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.416526 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:12.419548 (+ 3022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:12.420390 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:12.420400 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:12.421187 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:12.421691 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:12.421717 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.423314 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.423331 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:12.425164 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:12.425182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.426169 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.426179 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:12.426425 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:12.427107 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:12.427140 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:12.430169 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:12.434497 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:12.434506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:12.434515 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:12.434873 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:12.435291 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:12.435298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:12.435305 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:12.435472 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:12.436238 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:12.436244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:12.436420 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 16:05:12.436697 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 16:05:12.437125 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":341,"threads_started":1}
I20260924 16:05:12.438421 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.415666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.16.233.190:33939)
0924 16:05:12.418855 (+ 3189us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:12.418861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:12.418901 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:05:12.419902 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:12.419907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:12.420132 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:12.420285 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:12.421899 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.423152 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.425403 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.425916 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.426051 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:12.430474 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:12.430487 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:12.430505 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:12.430551 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:12.434373 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:12.435000 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:12.435006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:12.435008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:12.435145 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:12.435600 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:12.435604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:12.435606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:12.436106 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:12.436267 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:12.436926 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 16:05:12.437317 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":3010,"thread_start_us":2803,"threads_started":1}
I20260924 16:05:12.441416 8143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:05:12.442510 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.442019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35546)
0924 16:05:12.442224 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.442377 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:05:12.442750 8091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
I20260924 16:05:12.444378 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.443932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35548)
0924 16:05:12.444098 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.444195 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:05:12.444612 8143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:38739: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab.
Entry for principal kudu/127.16.233.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab.
I20260924 16:05:12.467778 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.189:38739
--webserver_interface=127.16.233.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab
--principal=kudu/127.16.233.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:33939,127.16.233.189:38739,127.16.233.188:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:12.494042 8143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } attempt: 1
I20260924 16:05:12.495297 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.494817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35562)
0924 16:05:12.495012 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.495134 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 16:05:12.496248 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.495921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35574)
0924 16:05:12.496058 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.496130 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:05:12.496441 8143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:38739: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
I20260924 16:05:12.564726 8143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } attempt: 2
I20260924 16:05:12.566009 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.565462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35586)
0924 16:05:12.565748 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.565858 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 16:05:12.567085 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.566694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35600)
0924 16:05:12.566866 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.566959 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 16:05:12.567330 8143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:38739: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
I20260924 16:05:12.658629 8143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } attempt: 3
I20260924 16:05:12.659817 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.659349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35604)
0924 16:05:12.659577 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.659669 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 16:05:12.660832 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.660473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35618)
0924 16:05:12.660632 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.660698 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 16:05:12.661037 8143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:38739: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
W20260924 16:05:12.787557 8150 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:05:12.787868 8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:12.787976 8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:12.788177 8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:12.795403 8150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:12.795492 8150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:12.795599 8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:12.795688 8150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:12.795811 8150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:12.804456 8143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } attempt: 4
I20260924 16:05:12.805640 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.805119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35630)
0924 16:05:12.805426 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.805512 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260924 16:05:12.805934 8150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:33939,127.16.233.189:38739,127.16.233.188:42775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.189.keytab
--principal=kudu/127.16.233.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.189:38739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.233.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:12.806784 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.806437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35642)
0924 16:05:12.806592 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.806668 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 16:05:12.806977 8143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.189:38739: Network error: Client connection negotiation failed: client connection to 127.16.233.189:38739: connect: Connection refused (error 111)
I20260924 16:05:12.807363 8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:12.808677 8150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:12.820329 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:12.820327 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:12.820582 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:12.821184 8150 server_base.cc:1034] running on GCE node
I20260924 16:05:12.821907 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:12.823364 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:12.824607 8150 hybrid_clock.cc:648] HybridClock initialized: now 1790265912824526 us; error 50 us; skew 500 ppm
I20260924 16:05:12.824925 8150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:12 dist-test-slave-xn4h krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:12.827875 8150 init.cc:377] Logged in from keytab as kudu/127.16.233.189@KRBTEST.COM (short username kudu)
I20260924 16:05:12.829226 8150 webserver.cc:460] Webserver started at http://127.16.233.189:37361/ using document root <none> and password file <none>
I20260924 16:05:12.829918 8150 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:12.830044 8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:12.830400 8150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:12.833242 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/data/instance:
uuid: "8320ea39cae647628604da9e00c802f1"
format_stamp: "Formatted at 2026-09-24 16:05:12 on dist-test-slave-xn4h"
server_key: "231c9f5afecbde30c6c1c48dfd652312"
server_key_iv: "c0c4dbbde4bdd3d3749e876cc7d80f8c"
server_key_version: "encryptionkey@0"
I20260924 16:05:12.834043 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/wal/instance:
uuid: "8320ea39cae647628604da9e00c802f1"
format_stamp: "Formatted at 2026-09-24 16:05:12 on dist-test-slave-xn4h"
server_key: "231c9f5afecbde30c6c1c48dfd652312"
server_key_iv: "c0c4dbbde4bdd3d3749e876cc7d80f8c"
server_key_version: "encryptionkey@0"
I20260924 16:05:12.839022 8150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 16:05:12.842389 8165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:12.843596 8150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:05:12.843773 8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/wal
uuid: "8320ea39cae647628604da9e00c802f1"
format_stamp: "Formatted at 2026-09-24 16:05:12 on dist-test-slave-xn4h"
server_key: "231c9f5afecbde30c6c1c48dfd652312"
server_key_iv: "c0c4dbbde4bdd3d3749e876cc7d80f8c"
server_key_version: "encryptionkey@0"
I20260924 16:05:12.843991 8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:12.874285 8150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:12.878173 8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:12.878481 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:12.896625 8150 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.189:38739
I20260924 16:05:12.896643 8216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.189:38739 every 8 connection(s)
I20260924 16:05:12.897903 8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/data/info.pb
I20260924 16:05:12.901472 8217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:12.902364 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 8150
I20260924 16:05:12.902575 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-1/wal/instance
I20260924 16:05:12.903060 17318 external_mini_cluster.cc:1428] Setting key 0936b570d4e1f41aecebeea7d74f0938
I20260924 16:05:12.908124 8217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:12 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.188@KRBTEST.COM; defaulting to no policy
I20260924 16:05:12.928057 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.910691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.16.233.190:33939)
0924 16:05:12.910926 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:12.910932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:12.912019 (+ 1087us) server_negotiation.cc:407] Connection header received
0924 16:05:12.913938 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:12.913944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:12.914034 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:12.914157 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:12.915614 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.916284 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.917742 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.918142 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.918288 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:12.922046 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:12.922051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:12.922055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:12.922086 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:12.924784 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:12.925689 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:12.925694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:12.925700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:12.925823 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:12.926287 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:12.926292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:12.926296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:12.926624 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:12.926797 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:12.927497 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 16:05:12.927722 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.188@KRBTEST.COM" created.
I20260924 16:05:12.929180 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.911202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:33939 (local address 127.0.0.1:60830)
0924 16:05:12.911858 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.911903 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:12.913652 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:12.914412 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:12.914420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:12.914927 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:12.915440 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:12.915457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.916447 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.916451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:12.917580 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:12.917593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.918383 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.918392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:12.918623 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:12.919333 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:12.919355 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:12.921806 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:12.924927 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:12.924936 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:12.924951 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:12.925577 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:12.925937 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:12.925942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:12.925946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:12.926178 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:12.926759 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:12.926767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:12.927141 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 16:05:12.927407 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 16:05:12.927937 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":253,"threads_started":1}
I20260924 16:05:12.930975 8217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:12 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:05:12.950281 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.931827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35656)
0924 16:05:12.932102 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.932122 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:12.932226 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:12.935381 (+ 3155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:12.935385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:12.935726 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:12.935856 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:12.935864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.937985 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.937993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:12.939471 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:12.939489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.940371 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.940380 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:12.940598 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:12.941213 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:12.941241 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:12.943663 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:12.947992 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:12.947997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:12.948000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:12.948409 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:12.948808 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:12.948811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:12.948813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:12.948953 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:12.949634 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:12.949637 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:12.949740 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 16:05:12.949851 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 16:05:12.950034 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260924 16:05:12.950805 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.931878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.16.233.189:38739)
0924 16:05:12.934842 (+ 2964us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:12.934852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:12.934885 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:05:12.934954 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:12.934965 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:12.935206 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:12.935379 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:12.936049 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.937832 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.939620 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:12.940199 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:12.940309 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:12.943881 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:12.943896 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:12.943924 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:12.943965 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:12.947838 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:12.948556 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:12.948566 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:12.948572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:12.948706 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:12.949055 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:12.949062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:12.949068 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:12.949505 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:12.949717 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:12.950023 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 16:05:12.950325 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2776,"thread_start_us":2621,"threads_started":1}
I20260924 16:05:12.953127 8217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:05:12.954245 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.953777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43128)
0924 16:05:12.953999 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.954112 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 16:05:12.954511 8166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
I20260924 16:05:12.956037 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:12.955702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43142)
0924 16:05:12.955873 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:12.955944 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:05:12.956257 8217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:42775: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab.
Entry for principal kudu/127.16.233.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab.
I20260924 16:05:12.963284 17318 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
/tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.188:42775
--webserver_interface=127.16.233.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab
--principal=kudu/127.16.233.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.148:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.190:33939,127.16.233.189:38739,127.16.233.188:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:13.005627 8217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } attempt: 1
I20260924 16:05:13.006880 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.006428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43154)
0924 16:05:13.006611 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.006722 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 16:05:13.007886 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.007578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43164)
0924 16:05:13.007711 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.007782 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:05:13.008114 8217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:42775: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
I20260924 16:05:13.076356 8217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } attempt: 2
I20260924 16:05:13.077605 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.077115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43174)
0924 16:05:13.077340 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.077446 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 16:05:13.078795 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.078438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43186)
0924 16:05:13.078604 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.078680 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 16:05:13.079130 8217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:42775: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
I20260924 16:05:13.106240 8143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } attempt: 5
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:05:13.131428 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.107009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35672)
0924 16:05:13.107251 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.107275 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:13.107442 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:13.107825 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:13.107833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:13.108357 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:13.108644 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.108662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.109697 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.109703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.111080 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.111097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.111904 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.111913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.112140 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.122357 (+ 10217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:13.122388 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:13.125183 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:13.128614 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.128619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.128624 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.129128 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.129472 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.129477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.129481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.129658 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.130298 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:13.130324 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:13.130510 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 16:05:13.130662 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 16:05:13.130987 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 16:05:13.131492 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.107116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.16.233.189:38739)
0924 16:05:13.107287 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.107291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.107362 (+ 71us) server_negotiation.cc:407] Connection header received
0924 16:05:13.107567 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.107571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.107644 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.107726 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.108880 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.109557 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.111271 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.111728 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.111842 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.125508 (+ 13666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.125514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.125517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.125553 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.128461 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.129242 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.129246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.129248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.129361 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.129828 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.129832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.129834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.130133 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.130353 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.130830 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.131099 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:05:13.132803 8143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:05:13.134049 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.133513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43196)
0924 16:05:13.133737 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.133862 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 16:05:13.135146 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.134697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43206)
0924 16:05:13.134899 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.134995 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 16:05:13.135370 8143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:42775: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
I20260924 16:05:13.170416 8217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } attempt: 3
I20260924 16:05:13.171670 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.171205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43208)
0924 16:05:13.171411 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.171503 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 16:05:13.172924 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.172411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43212)
0924 16:05:13.172709 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.172792 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260924 16:05:13.173168 8217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:42775: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
I20260924 16:05:13.186599 8143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } attempt: 1
I20260924 16:05:13.187696 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.187281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43218)
0924 16:05:13.187463 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.187550 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:05:13.188783 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.188437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43220)
0924 16:05:13.188591 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.188657 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 16:05:13.189003 8143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:42775: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
I20260924 16:05:13.257207 8143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } attempt: 2
I20260924 16:05:13.258446 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.257908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43226)
0924 16:05:13.258125 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.258214 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 16:05:13.259387 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.259076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43232)
0924 16:05:13.259216 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.259288 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:05:13.259572 8143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:42775: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
W20260924 16:05:13.292562 8224 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:05:13.292922 8224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:13.293056 8224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:13.293196 8224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:13.300213 8224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:13.300328 8224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:13.300433 8224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:13.300530 8224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:13.300618 8224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:13.310663 8224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.148:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.190:33939,127.16.233.189:38739,127.16.233.188:42775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/krb5kdc/kudu_127.16.233.188.keytab
--principal=kudu/127.16.233.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.188:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.233.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:13.311964 8224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:13.313253 8224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:05:13.316377 8217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } attempt: 4
I20260924 16:05:13.317571 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.317088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43242)
0924 16:05:13.317314 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.317418 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:05:13.318698 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.318206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43244)
0924 16:05:13.318407 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.318512 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 16:05:13.319018 8217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:42775: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
W20260924 16:05:13.325143 8231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:13.325229 8230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:13.325853 8233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:13.327534 8224 server_base.cc:1034] running on GCE node
I20260924 16:05:13.328198 8224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:13.329439 8224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:13.330667 8224 hybrid_clock.cc:648] HybridClock initialized: now 1790265913330598 us; error 27 us; skew 500 ppm
I20260924 16:05:13.330992 8224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265913, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:13.333989 8224 init.cc:377] Logged in from keytab as kudu/127.16.233.188@KRBTEST.COM (short username kudu)
I20260924 16:05:13.335338 8224 webserver.cc:460] Webserver started at http://127.16.233.188:45957/ using document root <none> and password file <none>
I20260924 16:05:13.335985 8224 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:13.336067 8224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:13.336326 8224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:13.339237 8224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/data/instance:
uuid: "2aec4a0a362e43609118c900c2f17683"
format_stamp: "Formatted at 2026-09-24 16:05:13 on dist-test-slave-xn4h"
server_key: "c3b112a772afbf5410d67815a28a8ba5"
server_key_iv: "0494d28363302f3437843bc15560d664"
server_key_version: "encryptionkey@0"
I20260924 16:05:13.340023 8224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/wal/instance:
uuid: "2aec4a0a362e43609118c900c2f17683"
format_stamp: "Formatted at 2026-09-24 16:05:13 on dist-test-slave-xn4h"
server_key: "c3b112a772afbf5410d67815a28a8ba5"
server_key_iv: "0494d28363302f3437843bc15560d664"
server_key_version: "encryptionkey@0"
I20260924 16:05:13.344792 8224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:05:13.348201 8240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:13.349498 8224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:05:13.349675 8224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/data,/tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/wal
uuid: "2aec4a0a362e43609118c900c2f17683"
format_stamp: "Formatted at 2026-09-24 16:05:13 on dist-test-slave-xn4h"
server_key: "c3b112a772afbf5410d67815a28a8ba5"
server_key_iv: "0494d28363302f3437843bc15560d664"
server_key_version: "encryptionkey@0"
I20260924 16:05:13.349905 8224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:13.365762 8143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } attempt: 3
I20260924 16:05:13.366895 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.366452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43256)
0924 16:05:13.366663 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.366748 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:05:13.367923 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.367522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43260)
0924 16:05:13.367737 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.367812 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260924 16:05:13.368135 8143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.188:42775: Network error: Client connection negotiation failed: client connection to 127.16.233.188:42775: connect: Connection refused (error 111)
I20260924 16:05:13.398702 8224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:13.403834 8224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:13.404115 8224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:13.424141 8224 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.188:42775
I20260924 16:05:13.424149 8291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.188:42775 every 8 connection(s)
I20260924 16:05:13.425477 8224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/data/info.pb
I20260924 16:05:13.426754 17318 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu as pid 8224
I20260924 16:05:13.426955 17318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRO1Jyr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265545293861-17318-0/minicluster-data/master-2/wal/instance
I20260924 16:05:13.427496 17318 external_mini_cluster.cc:1428] Setting key e99b388d5885957e3afc523f88a0a18f
I20260924 16:05:13.430197 8292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:13.436970 8292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265913, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
I20260924 16:05:13.449644 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.429333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.16.233.188:42775)
0924 16:05:13.430156 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.430168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.430213 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:05:13.431954 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.431976 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.432343 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.432737 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.434104 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.435500 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.436859 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.437605 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.437809 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.442467 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.442474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.442484 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.442517 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.445530 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.446203 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.446212 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.446225 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.446462 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.446837 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.446843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.446846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.447512 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.447838 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.448261 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.448680 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":319,"threads_started":1}
I20260924 16:05:13.454124 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.439661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.190:33939 (local address 127.0.0.1:60842)
0924 16:05:13.440426 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.440468 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:13.440647 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:13.441039 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:13.441046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:13.441704 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:13.441946 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.441954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.443059 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.443063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.444114 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.444126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.444865 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.444872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.445043 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.445641 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:13.445662 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:13.447816 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:13.451215 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.451223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.451228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.451621 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.452059 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.452063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.452067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.452224 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.452912 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:13.452919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:13.453131 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 16:05:13.453320 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 16:05:13.453633 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":420,"threads_started":1}
I20260924 16:05:13.454128 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.439564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.16.233.190:33939)
0924 16:05:13.440122 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.440131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.440563 (+ 432us) server_negotiation.cc:407] Connection header received
0924 16:05:13.440790 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.440795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.440857 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.440977 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.442091 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.442885 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.444263 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.444680 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.444824 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.448008 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.448012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.448016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.448046 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.451094 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.451743 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.451748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.451751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.451904 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.452349 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.452354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.452358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.452725 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.452890 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.453446 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.453681 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
I20260924 16:05:13.455937 8292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.190@KRBTEST.COM
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265913, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:05:13.469765 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.452850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.16.233.190:33939)
0924 16:05:13.453531 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.453538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.453562 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:05:13.453613 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.453616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.453667 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.453779 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.455377 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.456459 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.458161 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.458661 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.458795 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.463439 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.463445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.463448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.463477 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.466854 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.467465 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.467470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.467473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.467629 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.468025 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.468028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.468030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.468360 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.468592 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.468983 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.469210 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":332,"threads_started":1}
I20260924 16:05:13.470023 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.456585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.189:38739 (local address 127.0.0.1:35684)
0924 16:05:13.456744 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.456764 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:13.456852 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:13.457169 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:13.457173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:13.457617 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:13.457787 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.457798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.459262 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.459266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.460125 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.460137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.461227 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.461232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.461373 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.461947 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:13.461977 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:13.464079 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:13.467588 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.467592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.467594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.468103 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.468490 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.468495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.468498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.468665 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.469261 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:13.469266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:13.469419 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 16:05:13.469529 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:05:13.469713 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 16:05:13.470242 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.456676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.16.233.189:38739)
0924 16:05:13.456849 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.456853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.456869 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:05:13.456977 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.456982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.457039 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.457113 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.457934 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.458655 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.460281 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.461076 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.461216 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.464298 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.464306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.464312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.464352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.467458 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.468210 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.468216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.468218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.468369 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.468777 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.468780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.468782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.469114 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.469268 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.469649 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.469868 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:05:13.471304 8292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265913, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.188@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.189@KRBTEST.COM
I20260924 16:05:13.484428 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.472222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.16.233.188:42775)
0924 16:05:13.472395 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.472401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.472483 (+ 82us) server_negotiation.cc:407] Connection header received
0924 16:05:13.472643 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.472652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.472700 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.472792 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.473475 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.474171 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.475319 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.475776 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.475923 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.479336 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.479340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.479343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.479370 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.481861 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.482521 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.482528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.482532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.482650 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.483019 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.483024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.483027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.483284 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.483435 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.483785 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.484006 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:05:13.484493 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.472145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43278)
0924 16:05:13.472394 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.472420 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:13.472519 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:13.472806 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:13.472810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:13.473159 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:13.473330 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.473345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.474298 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.474344 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.475178 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.475189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.475971 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.475978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.476159 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.476809 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:13.476827 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:13.479142 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:13.481993 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.482000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.482004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.482414 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.482767 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.482782 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.482786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.482912 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.483396 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:13.483400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:13.483545 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 16:05:13.483686 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 16:05:13.484158 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":146}
I20260924 16:05:13.485975 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.471546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.16.233.189:38739)
0924 16:05:13.471688 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.471692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.471708 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:05:13.471890 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.471897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.471992 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.472090 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.473293 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.474048 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.475376 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.475761 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.475877 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.480528 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.480533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.480536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.480567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.483280 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.483925 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.483932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.483935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.484075 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.484456 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.484461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.484465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.484843 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.485134 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.485390 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.485626 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:05:13.489814 8292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683: Bootstrap starting.
I20260924 16:05:13.490876 17318 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:05:13.493314 8292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:13.494366 8292 log.cc:824] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:13.493162 8118 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "389b344632224d249eabd0208f7dae0c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60844
I20260924 16:05:13.497046 8292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683: No bootstrap required, opened a new log
I20260924 16:05:13.504012 8292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } }
I20260924 16:05:13.504318 8292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aec4a0a362e43609118c900c2f17683, State: Initialized, Role: FOLLOWER
I20260924 16:05:13.504747 8292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } }
I20260924 16:05:13.505597 8300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.505945 8300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:13.506824 8292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:05:13.516767 8311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:05:13.516892 8311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:05:13.513654 8224 master_runner.cc:186] Error getting master registration for 127.16.233.190:33939: OK, instance_id { permanent_uuid: "389b344632224d249eabd0208f7dae0c" instance_seqno: 1790265912395932 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:05:13.522655 8224 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:05:13.545300 8143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } attempt: 4
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.190@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
I20260924 16:05:13.560819 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.546119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43280)
0924 16:05:13.546356 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.546381 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:13.546523 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:13.546952 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:13.546957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:13.547423 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:13.547613 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.547622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.548698 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.548706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.549663 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.549675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.550506 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.550512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.550711 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.551375 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:13.551394 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:13.553722 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:13.557958 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.557964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.557968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.558526 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.558942 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.558947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.558950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.559111 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.559769 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:13.559779 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:13.559956 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 16:05:13.560118 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 16:05:13.560405 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 16:05:13.560851 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.546206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.16.233.188:42775)
0924 16:05:13.546435 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.546441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.546464 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:05:13.546671 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.546676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.546798 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.546916 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.547764 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.548560 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.549853 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.550356 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.550495 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.553966 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.553974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.553979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.554027 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.557795 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.558655 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.558662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.558671 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.558813 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.559236 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.559241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.559245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.559630 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.559805 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.560265 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.560483 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:05:13.568141 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c: Bootstrap starting.
I20260924 16:05:13.572364 8143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:13.573629 8143 log.cc:824] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:13.576714 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c: No bootstrap required, opened a new log
I20260924 16:05:13.581815 8143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } }
I20260924 16:05:13.582165 8143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389b344632224d249eabd0208f7dae0c, State: Initialized, Role: FOLLOWER
I20260924 16:05:13.582617 8143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } }
I20260924 16:05:13.583546 8314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.583971 8314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:13.584666 8143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:13.584805 8118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:05:13.585057 8118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:13.589233 8118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } }
I20260924 16:05:13.591774 8118 leader_election.cc:288] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [CANDIDATE]: Term 1 election: Requested vote from peers 8320ea39cae647628604da9e00c802f1 (127.16.233.189:38739), 2aec4a0a362e43609118c900c2f17683 (127.16.233.188:42775)
I20260924 16:05:13.593864 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "389b344632224d249eabd0208f7dae0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2aec4a0a362e43609118c900c2f17683"
I20260924 16:05:13.594368 8267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:13.598982 8267 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 389b344632224d249eabd0208f7dae0c in term 1.
I20260924 16:05:13.600190 8090 leader_election.cc:302] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aec4a0a362e43609118c900c2f17683, 389b344632224d249eabd0208f7dae0c; no voters:
I20260924 16:05:13.600826 8314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:13.603811 8314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [term 1 LEADER]: Becoming Leader. State: Replica: 389b344632224d249eabd0208f7dae0c, State: Running, Role: LEADER
I20260924 16:05:13.604570 8314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } }
I20260924 16:05:13.616874 8329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389b344632224d249eabd0208f7dae0c. Latest consensus state: current_term: 1 leader_uuid: "389b344632224d249eabd0208f7dae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.617342 8329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:13.618220 8217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } attempt: 5
W20260924 16:05:13.633183 8326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:05:13.633337 8326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:05:13.634423 8333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 16:05:13.635814 8095 tablet.cc:2369] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:13.636797 8333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:13.618703 8074 master_runner.cc:186] Error getting master registration for 127.16.233.189:38739: OK, instance_id { permanent_uuid: "8320ea39cae647628604da9e00c802f1" instance_seqno: 1790265912891515 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:05:13.637276 8074 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 16:05:13 dist-test-slave-xn4h krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.189@KRBTEST.COM for kudu/127.16.233.188@KRBTEST.COM
I20260924 16:05:13.647239 8267 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [term 1 FOLLOWER]: Refusing update from remote peer 389b344632224d249eabd0208f7dae0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:13.648303 8329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:13.650247 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.626471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.16.233.188:42775)
0924 16:05:13.626669 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.626675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.626696 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:05:13.626762 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.626768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.626866 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.626981 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.628425 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.629886 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.631843 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.632545 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.632685 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.640481 (+ 7796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.640488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.640493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.640536 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.644420 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.645686 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.645694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.645700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.645980 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.646825 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.646831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.646834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.647620 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.647904 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.648934 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.649444 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:05:13.651301 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.618740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.188:42775 (local address 127.0.0.1:43296)
0924 16:05:13.618929 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:13.618955 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:13.619090 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:13.627048 (+ 7958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:13.627056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:13.627732 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:13.628107 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.628126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.630120 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.630129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.631544 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:13.631562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.632942 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.632948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:13.633195 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.634216 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:13.634242 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:13.640131 (+ 5889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:13.644648 (+ 4517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.644656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.644663 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.645485 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.646279 (+ 794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:13.646284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:13.646288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:13.646628 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:13.648031 (+ 1403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:13.648036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:13.648326 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 16:05:13.649400 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0924 16:05:13.650154 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 16:05:13.656454 8246 tablet.cc:2369] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:13.672989 8300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389b344632224d249eabd0208f7dae0c. Latest consensus state: current_term: 1 leader_uuid: "389b344632224d249eabd0208f7dae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.673367 8300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:13.677995 8217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1: Bootstrap starting.
I20260924 16:05:13.681993 8217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:13.683669 8217 log.cc:824] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:13.688675 8217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1: No bootstrap required, opened a new log
I20260924 16:05:13.689769 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.640925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.16.233.189:38739)
0924 16:05:13.641089 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.641096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.652682 (+ 11586us) server_negotiation.cc:407] Connection header received
0924 16:05:13.653111 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.653116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.653194 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.653302 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.658518 (+ 5216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.660033 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.664981 (+ 4948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.665816 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.665912 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.669724 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.669733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.669739 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.669780 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.673603 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.683821 (+ 10218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.683828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.683831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.684043 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.685548 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.685554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.685557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.686131 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.686463 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.688295 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.688618 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 16:05:13.699864 8300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "389b344632224d249eabd0208f7dae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.700155 8300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:13.702117 8217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } }
I20260924 16:05:13.702545 8217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8320ea39cae647628604da9e00c802f1, State: Initialized, Role: FOLLOWER
I20260924 16:05:13.703049 8217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } }
I20260924 16:05:13.703965 8329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "389b344632224d249eabd0208f7dae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.704331 8329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:13.706125 8333 catalog_manager.cc:1347] Generated new cluster ID: f4e799d188f8494488786c658d3b2820
I20260924 16:05:13.706219 8333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:13.714973 8345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.715454 8345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:13.716670 8217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:13.717206 8191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:13.722508 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "389b344632224d249eabd0208f7dae0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8320ea39cae647628604da9e00c802f1"
I20260924 16:05:13.722286 8191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [term 1 FOLLOWER]: Refusing update from remote peer 389b344632224d249eabd0208f7dae0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:13.728351 8192 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 389b344632224d249eabd0208f7dae0c in term 1.
I20260924 16:05:13.731870 8329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:13.784612 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.664811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.16.233.190:33939)
0924 16:05:13.665020 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.665025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.681266 (+ 16241us) server_negotiation.cc:407] Connection header received
0924 16:05:13.683562 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.683567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.683679 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.683769 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.699546 (+ 15777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.700751 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.754681 (+ 53930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.755717 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.755867 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.774607 (+ 18740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.774620 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.774628 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.774701 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.779869 (+ 5168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.781232 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.781241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.781248 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.781470 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.782072 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.782081 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.782087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.782615 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.782953 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.783738 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.784090 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:05:13.793464 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:13.643808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.16.233.188:42775)
0924 16:05:13.650567 (+ 6759us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.650578 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.681478 (+ 30900us) server_negotiation.cc:407] Connection header received
0924 16:05:13.683043 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.683057 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.683190 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.683274 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.697500 (+ 14226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.698636 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.750075 (+ 51439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:13.751131 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:13.751287 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:13.785782 (+ 34495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:13.785791 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:13.785800 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:13.785858 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:13.789906 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.790846 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.790852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.790855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.791018 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:13.791506 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:13.791511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:13.791514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:13.791895 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:13.792194 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:13.792770 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 16:05:13.793030 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6605,"thread_start_us":27894,"threads_started":1}
I20260924 16:05:13.810583 8345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389b344632224d249eabd0208f7dae0c. Latest consensus state: current_term: 1 leader_uuid: "389b344632224d249eabd0208f7dae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.810987 8345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:13.811260 8357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "389b344632224d249eabd0208f7dae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.811647 8357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:05:13.820649 8361 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:05:13.822772 8314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "389b344632224d249eabd0208f7dae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389b344632224d249eabd0208f7dae0c" member_type: VOTER last_known_addr { host: "127.16.233.190" port: 33939 } } peers { permanent_uuid: "8320ea39cae647628604da9e00c802f1" member_type: VOTER last_known_addr { host: "127.16.233.189" port: 38739 } } peers { permanent_uuid: "2aec4a0a362e43609118c900c2f17683" member_type: VOTER last_known_addr { host: "127.16.233.188" port: 42775 } } }
I20260924 16:05:13.823107 8314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:13.832587 8333 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:13.834596 8333 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:13.850956 8333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 389b344632224d249eabd0208f7dae0c: Generated new TSK 0
I20260924 16:05:13.852144 8333 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:05:13.963096 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:13.945658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.16.233.190:33939)
0924 16:05:13.946200 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.946210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.960227 (+ 14017us) server_negotiation.cc:407] Connection header received
0924 16:05:13.960825 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.960834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.960993 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.961096 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.962736 (+ 1640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60850: BlockingRecv error: recv got EOF from 127.0.0.1:60850 (error 108)
Metrics: {"server-negotiator.queue_time_us":360}
W20260924 16:05:13.966044 8295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:13.946416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.16.233.188:42775)
0924 16:05:13.946890 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.946899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.964082 (+ 17183us) server_negotiation.cc:407] Connection header received
0924 16:05:13.964346 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.964355 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.964471 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.964562 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.965734 (+ 1172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43318: BlockingRecv error: recv got EOF from 127.0.0.1:43318 (error 108)
Metrics: {"server-negotiator.queue_time_us":310}
W20260924 16:05:13.968611 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:13.946638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.16.233.189:38739)
0924 16:05:13.946868 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.946874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.966771 (+ 19897us) server_negotiation.cc:407] Connection header received
0924 16:05:13.967034 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.967042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.967160 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.967240 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.968306 (+ 1066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35698: BlockingRecv error: recv got EOF from 127.0.0.1:35698 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 16:05:13.976078 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:13.974218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.16.233.189:38739)
0924 16:05:13.974421 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.974427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.974584 (+ 157us) server_negotiation.cc:407] Connection header received
0924 16:05:13.974866 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.974872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.974970 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.975085 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.975800 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35708: BlockingRecv error: recv got EOF from 127.0.0.1:35708 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 16:05:13.982352 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:13.980189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35718 (local address 127.16.233.189:38739)
0924 16:05:13.980403 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.980412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.980964 (+ 552us) server_negotiation.cc:407] Connection header received
0924 16:05:13.981165 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.981172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.981243 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.981346 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.982141 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35718: BlockingRecv error: recv got EOF from 127.0.0.1:35718 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 16:05:13.990335 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:13.988268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35728 (local address 127.16.233.189:38739)
0924 16:05:13.988469 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.988476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.988953 (+ 477us) server_negotiation.cc:407] Connection header received
0924 16:05:13.989171 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.989177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.989271 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.989376 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.990073 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35728: BlockingRecv error: recv got EOF from 127.0.0.1:35728 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 16:05:13.999926 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:13.997632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.16.233.189:38739)
0924 16:05:13.997868 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:13.997879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:13.998473 (+ 594us) server_negotiation.cc:407] Connection header received
0924 16:05:13.998700 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:13.998707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:13.998809 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:13.998923 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:13.999653 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35732: BlockingRecv error: recv got EOF from 127.0.0.1:35732 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 16:05:14.013613 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:14.011401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.16.233.189:38739)
0924 16:05:14.011665 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:14.011673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:14.012218 (+ 545us) server_negotiation.cc:407] Connection header received
0924 16:05:14.012450 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:14.012454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:14.012529 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:14.012641 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:14.013374 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35748: BlockingRecv error: recv got EOF from 127.0.0.1:35748 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 16:05:14.035362 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:14.033056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.16.233.189:38739)
0924 16:05:14.033288 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:14.033298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:14.033871 (+ 573us) server_negotiation.cc:407] Connection header received
0924 16:05:14.034103 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:14.034109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:14.034192 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:14.034307 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:14.035124 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35750: BlockingRecv error: recv got EOF from 127.0.0.1:35750 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 16:05:14.073232 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:14.070932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.16.233.189:38739)
0924 16:05:14.071197 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:14.071207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:14.071734 (+ 527us) server_negotiation.cc:407] Connection header received
0924 16:05:14.071974 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:14.071979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:14.072061 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:14.072174 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:14.072963 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35754: BlockingRecv error: recv got EOF from 127.0.0.1:35754 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260924 16:05:14.150085 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:14.147513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.16.233.189:38739)
0924 16:05:14.147808 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:14.147815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:14.148158 (+ 343us) server_negotiation.cc:407] Connection header received
0924 16:05:14.148485 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:14.148492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:14.148596 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:14.148716 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:14.149853 (+ 1137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35758: BlockingRecv error: recv got EOF from 127.0.0.1:35758 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 16:05:14.286509 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:14.284091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.16.233.189:38739)
0924 16:05:14.284268 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:14.284279 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:14.284634 (+ 355us) server_negotiation.cc:407] Connection header received
0924 16:05:14.284841 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:14.284849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:14.284962 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:14.285112 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:14.286151 (+ 1039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35762: BlockingRecv error: recv got EOF from 127.0.0.1:35762 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:05:14.518981 8311 catalog_manager.cc:1259] Loaded cluster ID: f4e799d188f8494488786c658d3b2820
I20260924 16:05:14.519078 8311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683: loading cluster ID for follower catalog manager: success
I20260924 16:05:14.521538 8311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683: acquiring CA information for follower catalog manager: success
I20260924 16:05:14.523344 8311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2aec4a0a362e43609118c900c2f17683: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 16:05:14.548352 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:14.546612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35774 (local address 127.16.233.189:38739)
0924 16:05:14.546798 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:14.546802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:14.547068 (+ 266us) server_negotiation.cc:407] Connection header received
0924 16:05:14.547292 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:14.547296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:14.547407 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:14.547500 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:14.548125 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35774: BlockingRecv error: recv got EOF from 127.0.0.1:35774 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:05:14.826500 8361 catalog_manager.cc:1259] Loaded cluster ID: f4e799d188f8494488786c658d3b2820
I20260924 16:05:14.826654 8361 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1: loading cluster ID for follower catalog manager: success
I20260924 16:05:14.829367 8361 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1: acquiring CA information for follower catalog manager: success
I20260924 16:05:14.831349 8361 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8320ea39cae647628604da9e00c802f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:05:15.071573 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:15.064126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.16.233.189:38739)
0924 16:05:15.064925 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:15.064929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:15.064946 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:05:15.065009 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:15.065012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:15.065163 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:15.065266 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:15.065830 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.066788 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.068811 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.069285 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.069388 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:15.070061 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:15.070744 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:15.070872 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:15.071034 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 16:05:15.071115 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":517,"threads_started":1}
I20260924 16:05:15.085143 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:15.078300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.16.233.188:42775)
0924 16:05:15.079011 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:15.079015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:15.079035 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:05:15.079137 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:15.079141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:15.079306 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:15.079449 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:15.079967 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.080692 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.082691 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.083108 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.083216 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:15.083835 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:15.084371 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:15.084482 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:15.084622 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 16:05:15.084751 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":466,"threads_started":1}
I20260924 16:05:15.102109 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:15.091603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.16.233.189:38739)
0924 16:05:15.091832 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:15.091838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:15.093068 (+ 1230us) server_negotiation.cc:407] Connection header received
0924 16:05:15.093392 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:15.093401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:15.093524 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:15.093602 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:15.094523 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.096027 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.098936 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.099911 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.100036 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:15.100945 (+ 909us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:15.101138 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:15.101223 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:15.101507 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 16:05:15.101627 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:05:15.109575 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:15.092863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.16.233.188:42775)
0924 16:05:15.093048 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:15.093054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:15.102914 (+ 9860us) server_negotiation.cc:407] Connection header received
0924 16:05:15.103183 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:15.103188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:15.103297 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:15.103361 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:15.104525 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.105526 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.107440 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.107865 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.108055 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:15.108593 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:15.108702 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:15.108808 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:15.109163 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:05:15.109254 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:05:15.111891 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:15.092375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.16.233.190:33939)
0924 16:05:15.102695 (+ 10320us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:15.102702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:15.102723 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:05:15.102849 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:15.102858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:15.103113 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:15.103293 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:15.105234 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.106369 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.108630 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:15.109349 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:15.110013 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:15.110661 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:15.111163 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:15.111287 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:15.111503 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 16:05:15.111594 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10181,"thread_start_us":11043,"threads_started":1}
I20260924 16:05:15.116991 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 8074
I20260924 16:05:15.118125 8138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:15.234575 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 8074
I20260924 16:05:15.256408 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 8150
I20260924 16:05:15.257596 8212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:15.365468 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 8150
I20260924 16:05:15.385888 17318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu pid 8224
I20260924 16:05:15.387012 8287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:15.492167 17318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRO1Jyr/build/asan/bin/kudu with pid 8224
2026-09-24T16:05:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4898 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9264 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (369692 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST